Agentic AI and multimodal prompt injection: why autonomous agents face a larger attack surface than chat models

1. What makes agentic systems structurally different

The term "agentic AI" covers a wide range of system architectures, but the features that create distinct security requirements relative to standard chat deployments are consistent:

Tool access with real-world side effects

Agentic systems are defined by their ability to take actions: write to a file system, make API calls, send messages, execute code, browse the web, control a computer's graphical interface. In a chat model deployment, the worst outcome of a successful prompt injection is a bad text response — incorrect information, an inappropriate reply, a disclosed system prompt. In an agentic deployment, the worst outcome is an autonomous series of real-world actions taken under the attacker's direction. The model is not just talking differently; it is doing different things.

This difference is what OWASP LLM06:2025 Excessive Agency formalises: the risk class that arises when an LLM with excessive tool access is compromised. The multimodal dimension of LLM06 specifically covers the case where the compromise arrives via a pixel-domain payload that text-only monitoring never sees.

Multiple image input boundaries per session

In a standard multimodal chat deployment, there is one image input event per turn: the user attaches an image, the model processes it, the model responds. Scanning at that boundary covers the full image attack surface for that turn. Agentic systems introduce additional image input events throughout the session lifecycle:

• Retrieval-time images. An agent that performs RAG lookups may retrieve documents that contain embedded images. The document was ingested into the vector store at some earlier time; the agent fetches it now, mid-session, and the VLM reads the image content as retrieved context. This is a distinct input boundary from the user-facing entry point.
• Mid-loop screenshot capture. Computer-use agents take screenshots during execution — to observe the current state of the browser or desktop before deciding their next action. The screen content is attacker-controlled if the agent is browsing an adversarial page. A scanner at the user input boundary saw the benign initial instruction; it does not see the adversarial screen content the agent photographed 10 tool calls later.
• Tool outputs containing images. Web-browsing agents receive rendered page content. Vision-enabled code-review agents may receive diagram outputs from code execution. File-system agents may read image files as part of their task. Each of these is a new image input that enters the agent's context outside of any user-facing scanning boundary.
• Subagent outputs in multi-agent systems. When one agent delegates to another, the subagent's response re-enters the orchestrator's context as trusted intermediate output. If the subagent processed an injected image and its behaviour was redirected, the injection can propagate upward via the subagent's text output — a trust-escalation vector covered in section 3 below.

None of these input events is visible to a scanner positioned at the user-facing front door. A text-only scanner positioned there has even less coverage: it never saw the images, and now it also does not see the in-loop injection points. The coverage argument for multimodal scanning in agent systems is therefore even stronger than in chat deployments — there are more attack surfaces, not fewer.

Persistent state that carries contamination

Many agentic systems maintain persistent state between turns: a scratchpad, a memory store, a conversation history buffer. A successful injection that redirects the agent's behaviour in one turn can write to this persistent state — a self-reinforcing mechanism where the attacker's instruction survives beyond the turn where it was injected and continues to influence subsequent turns. A text-only monitor on the conversation history cannot identify this contamination if the original payload arrived in the pixel layer of an image that was never scanned.

2. Three attack chains that do not exist in non-agentic deployments

Attack chain 1: Computer-use agent via adversarial screen content

A computer-use agent — such as a system built on Anthropic's computer use API, GPT-4o with function calling over screenshot loops, or an open-source computer-use framework — operates by taking screenshots of the current desktop or browser state and deciding what action to take next. The agent's perception is visual: it reads the screen as a series of images and interprets them with a VLM.

An attacker who can control what appears on the screen the agent is browsing controls the agent's visual input. A web page that displays adversarial text — formatted in a font that appears innocuous to a human viewer but reads as a FigStep-class instruction to the VLM — can redirect the agent's tool calls. The agent follows the instruction it reads from the pixel layer, using the tools it has access to (browser navigation, file downloads, clipboard paste, form submission), and the text-only monitor on the agent's conversation history sees only the benign action log: "navigated to page", "clicked element", "submitted form".

The specific vulnerability profile of computer-use agent deployments — CSS-invisible overlays, navigation-event injection, terminal output hijacking — is detailed at computer-use agent prompt injection scanning. The same pixel-domain attack mechanics used in FigStep apply directly to this surface; the scanner must run on the raw screenshot bytes before the VLM call that interprets the screen state.

Attack chain 2: Persistent corpus poisoning in agentic RAG

A RAG-augmented agent that retrieves context from a document corpus faces a distinct attack geometry from a chat model that receives an image from a live user. In the chat model case, the attacker must deliver their payload to a specific user's session in real time. In the RAG agent case, a single successfully ingested adversarial document delivers its payload to every agent query that retrieves it — potentially across many sessions, many users, and an unbounded future time window.

The attack works at two layers simultaneously. The adversarial image in the document carries a FigStep-class pixel instruction that the VLM executes when it reads the retrieved context. At the same time, the document's CLIP-space embedding is crafted to rank highly on retrieval for the target query terms — ensuring the payload document surfaces in every relevant retrieval, not just occasionally. This is the OWASP LLM08:2025 vector embedding weakness applied to image payloads: the adversarial image crafts its embedding so that retrieval delivers it exactly when the attacker wants it.

A scanner positioned at the document ingestion pipeline catches this attack before the document enters the corpus. A scanner positioned only at the user input boundary will never see a retrieved document: the image arrives in the agent's context from the retrieval system, not from the user. The agentic RAG pipeline injection page details the specific scan placement, the LangChain and LlamaIndex integration points, and the per-document scan_id provenance record that satisfies SOC 2 CC6.6 evidence requirements for third-party data ingestion controls.

Attack chain 3: Trust escalation in multi-agent orchestration

Many production agentic systems use a multi-agent architecture: an orchestrator agent coordinates the overall task and delegates subtasks to specialised worker agents. The orchestrator receives worker outputs as internal context and treats them with higher trust than direct user input — because they are, nominally, the system's own intermediate reasoning rather than external attacker-controlled input.

This trust model creates a propagation vector for multimodal injection. A worker agent that is assigned to process documents, review code, or interact with visual interfaces receives image inputs as part of its task. If one of those images contains a FigStep or AgentTypo payload, the worker agent's behaviour is redirected. Critically, the worker's text output — which the orchestrator reads as trusted context — will reflect the redirected instruction. The orchestrator, which may never process images itself, receives the attacker's instruction embedded in what it believes is its own subagent's analysis.

The defence implication is that scanning cannot be confined to the orchestrator's user-facing input: every worker agent's image inputs require multimodal scanning, because a compromised worker is a compromised orchestrator. The full multi-agent trust model and how scanning placement must account for it is in the screenshot-reading agent injection page, which covers the related case of agents that delegate visual interpretation to specialised vision subagents.

3. Why text-only scanners are particularly dangerous in the agentic context

The structural blindness of text-only scanners to pixel-domain payloads is the same regardless of whether the deployment is a chat model or an autonomous agent. What changes is the consequence of deploying a scanner with that structural blindness in an agentic context.

False confidence at a non-representative input boundary

A text-only scanner placed at the user-facing entry point of an agent provides coverage of exactly one of the many image input boundaries the agent has. It provides no coverage of retrieval-time images, mid-loop screenshots, tool outputs, or subagent propagation. But its presence creates a documented control — "we have a prompt injection scanner on user inputs" — that can be cited in security reviews, SOC 2 audits, and compliance assessments. That documented control is accurate about what it covers and materially silent about what it does not cover, which in an agentic system is the majority of the attack surface.

Teams that have deployed text-only scanners for their agentic systems and cite that deployment as their PI defence posture have a much larger unscanned attack surface than teams running a standalone chat model with the same scanner. The agentic deployment is harder to secure, not easier, and a scanner that was adequate for the chat model is inadequate for the agent by construction.

The forensic dead end

When a conventional application is compromised, forensic analysis can identify the payload — the HTTP request that contained the SQL injection, the email that carried the malware attachment, the input that triggered the buffer overflow. The payload is in the log.

When an agentic system is compromised by a multimodal injection that was never scanned, the forensic record contains only the tool calls the agent made: "retrieved document ID 4721", "navigated to URL", "wrote to file /data/export.json". The pixel payload that triggered those actions was in the image bytes of document ID 4721 or the screenshot taken during URL navigation. If those bytes were never scanned and no scan record was created, the forensic trail ends at the tool call log. There is no artifact that shows what the VLM read from the pixel layer. Reconstructing the attack requires retrieving the original image bytes and running a scanner retroactively — which requires having preserved the raw bytes in a queryable audit log, a step that most agent frameworks do not implement by default.

Per-request scan records — scan_id, risk_score, flagged_region, image_hash — are the forensic artifact that makes this reconstruction possible without having to retrieve and re-run every image the agent ever processed. This is the audit evidence requirement that EU AI Act Article 15 cybersecurity controls and OWASP LLM Top 10 recommend for AI system input validation, and it only exists if a scanner runs on the raw bytes at the time of processing.

The OWASP LLM06 intersection

OWASP LLM06:2025 Excessive Agency defines the risk class that arises when an LLM with broad tool access is redirected by an injected instruction. Most LLM06 discussions focus on the text injection vector — a malicious system prompt or user message that instructs the agent to misuse its tools. The multimodal dimension of LLM06 is the case where the redirection arrives via a pixel-domain payload, which means the standard text-monitoring controls that LLM06 mitigations typically recommend are insufficient: you cannot monitor for the injected instruction if it was never in the text layer.

The combined LLM01 (prompt injection) plus LLM06 (excessive agency) risk — where the injection vector is multimodal and the consequence is unrestricted tool execution — is the highest-consequence risk class for agentic AI deployments. It is also the risk class for which text-only security tooling provides the least coverage, because the entry point (image or audio) and the consequence amplifier (agent tool access) both lie outside the scope of string-based classifiers. The multimodal LLM06 excessive agency page covers the specific scan integration patterns for LangGraph, CrewAI, and AutoGen-based agentic frameworks.

4. Defence architecture for agentic multimodal deployments

Securing an agentic system against multimodal prompt injection requires extending the three-layer defence stack described in the FigStep/AgentTypo/WhisperInject post to account for the multiple image input boundaries unique to agentic loops.

Scan at every image boundary in the agent loop, not just the entry point

The minimum scanning requirement for an agentic system is one multimodal scan at every point where image bytes enter the agent's context:

• User input boundary. Same as for chat models: scan raw image bytes before the first VLM call. This is the boundary text scanners are typically placed at, and it is necessary but not sufficient for agents.
• Retrieval boundary. Scan image content in retrieved documents before passing retrieved context to the agent. For LangChain-based RAG, this means a scan step in the retrieval chain between the retriever and the context formatter. For LlamaIndex, a scan node in the query pipeline before the synthesiser. The agentic RAG pipeline injection page provides the specific node insertion patterns for each framework.
• Screenshot boundary. For computer-use agents, scan each screenshot before passing it to the VLM action decision call. In a screenshot-action loop, this means a scan gate on every iteration: screenshot → scan → if risk score below threshold, pass to VLM; else halt and surface to human review. The latency cost of scanning a 1 MB screenshot at Glyphward's p95 under 200ms is typically acceptable within a loop step that already includes a VLM call; the scan adds less latency than the model call it gates.
• Subagent output boundary. In multi-agent orchestration, scan any text output from a subagent that processed images before routing it to the orchestrator as trusted context. This is the propagation vector for trust-escalation attacks and is the boundary most commonly left unscanned in multi-agent implementations.

Human-in-the-loop escalation on high-risk scores

For agentic systems with irreversible tool access (file writes, external API calls, messages sent), a scan score above a high-risk threshold should trigger a human-in-the-loop checkpoint before the next tool call executes. In LangGraph-based agents, this is a conditional interrupt: if the scan gate node returns a score above the configured threshold, the graph pauses at the next node and surfaces the flagged image and score to an operator review queue before resuming. This is the agentic analogue of the fail-closed pattern for chat models; in the agent context, fail-closed means "pause and await human confirmation" rather than "return an error to the user", because the agent has ongoing state that should not be abandoned mid-task without operator visibility.

Per-action audit records linking scan_id to tool call

The forensic gap described in section 3 is closed by linking each scan record to the subsequent tool call it gated. A per-action audit record contains: scan_id (from the multimodal scanner), image_hash (SHA-256 of the raw bytes scanned), risk_score at scan time, the tool call that followed (action name, parameters), and a timestamp. This record can be written to an append-only audit log independently of whether the scan flagged anything — clean scans create records too, which is what enables retroactive forensic reconstruction if a novel attack variant is later discovered in the payload corpus and you need to determine whether any prior session was exposed.

This audit architecture is also the evidence format that satisfies NIST AI RMF MAP 5.2 adversarial-input logging requirements, SOC 2 CC7.2 anomaly monitoring controls for AI systems, and EU AI Act Article 15 robustness-and-security record-keeping — all of which require documented evidence that inputs to high-risk AI systems were validated, not just an assertion that a scanner was deployed.

The scan placement decision: real-time vs. pre-ingestion

For retrieval-time images, there is a choice between scanning at ingestion (before the document enters the corpus) and scanning at retrieval (when the document is fetched for a specific agent query). Both have advantages. Ingestion-time scanning catches poisoned documents before they can be retrieved by any session; retrieval-time scanning catches documents that were clean when ingested but whose payload was only recognised after the payload corpus was updated. The real-time vs. batch scanning decision guide covers this trade-off in detail; for high-assurance agent deployments, the standard recommendation is both — ingestion gate to prevent corpus poisoning, retrieval gate to catch anything the ingestion scan missed with an older payload signature set.

5. Practical starting point for a production agentic deployment

If you are shipping an agentic system that accepts image inputs — document processing agents, computer-use automation, screenshot-reading assistants, voice agents with image tools, multi-agent pipelines with visual subagents — the minimum viable multimodal security posture involves three concrete changes to your current architecture:

1. Audit your image input boundaries. Map every point in your agent loop where image bytes enter the agent's context. This is not just the user-facing entry point; include retrieval steps, screenshot capture points, tool outputs, and any inter-agent message passing. The audit is usually a two-hour exercise for a well-understood agent graph. What it typically reveals is three to five additional image input boundaries that no current scanning covers.
2. Place a multimodal scan gate at each boundary. At each identified boundary, add a pre-VLM-call scan step using a scanner that operates on raw image bytes. The scan should return a risk score and a flagged region. Set a threshold appropriate to the action the agent takes if the scan passes — lower thresholds for tool calls with irreversible consequences, higher thresholds for read-only operations. For teams evaluating this, Glyphward's free tier (10 scans/day, no card required) at Glyphward pricing covers evaluation of all identified boundaries including screenshot loops.
3. Implement per-scan audit records. Write scan_id, image_hash, risk_score, and the subsequent tool call to an append-only log for every scan, regardless of score. This is a one-time implementation cost; it is the difference between a security posture you can evidence in an audit and one you can only assert.

For teams building on specific frameworks, the integration patterns for LangGraph (LangChain), AutoGen, and CrewAI are covered in the agentic RAG pipeline injection page; for Amazon Bedrock Agents specifically, see Bedrock Agents scanning; for coding assistant agents that process design mockups and screenshot context, see AI coding assistant context injection. The vision-language model security page covers the underlying VLM architecture argument for why the pixel layer is structurally the only position from which these attacks can be detected, independent of the agent framework layered above it.

FAQ

Further reading

• Computer-use agent prompt injection scanning — CSS-invisible overlay attacks, navigation-event injection, terminal output hijacking, and fail-closed screenshot loop integration for Anthropic computer use and GPT-4o computer use architectures.
• Agentic RAG pipeline injection — retrieval-boundary scan placement, LangChain and LlamaIndex integration, multi-hop injection propagation taxonomy, and the per-document scan_id provenance record for SOC 2 CC6.6 compliance.
• Screenshot-reading agent injection — visual-delegation trust model in multi-agent systems, scan placement in screenshot-based agent loops, and the specific attack geometry where the agent's perception (screenshot) is controlled by the attacker (screen content).
• OWASP LLM06:2025 Excessive Agency — multimodal — the specific intersection of tool-access risk amplification and pixel-domain injection; LangGraph HITL checkpoint integration; and the combined LLM01 + LLM06 risk taxonomy for agentic deployments.
• OWASP LLM08:2025 Vector and Embedding Weaknesses — multimodal — how adversarial CLIP embeddings craft retrieval rank for payload delivery; ingestion-time vs. retrieval-time scan placement; and the corpus poisoning persistence mechanism that makes RAG agents uniquely vulnerable.
• Real-time vs. batch prompt injection scanning — the architecture decision guide for ingestion-gate vs. retrieval-gate vs. both; latency budgets for inline scanning in agent loops; and the audit trail requirements that apply regardless of scan placement.
• AI coding assistant context injection — the specific case of coding agents (Cursor, GitHub Copilot Workspace, Codeium) that process design mockups, architecture diagrams, and screenshot context in their multi-file agent loops.
• AWS Bedrock Agents scanning — scan gate integration for Bedrock Agents Knowledge Base retrieval and ActionGroup tool calls; why Bedrock Guardrails text-coverage scope does not extend to image inputs in agent sessions.
• Vision-language model security — the VLM architecture argument for why pixel-layer scanning is the only placement with coverage for all image-domain injection attacks, independent of the agent framework or orchestration layer above it.
• FigStep, AgentTypo, WhisperInject — the three attacks every text scanner gets wrong — the foundational attack mechanics that apply to every multimodal deployment, agentic or not; covers why text scanners are structurally blind at the pixel and waveform layers.
• Why every text-only scanner misses a 30-pixel PNG — the architectural argument for the scanner coverage gap: what the VLM sees vs. what the text scanner sees, and why increasing text scanner sensitivity does not close the image-domain gap.
• Glyphward pricing — free tier (10 scans/day, no card) covers multimodal scanning across all agent loop boundaries including screenshot loops and retrieval gates. Pro at $29/mo adds webhook delivery, SDK, and per-request audit log export.