Prompt injection in autonomous AI research agents

The four multimodal injection surfaces in AI research agents

1. Web search result images. Research agents query Google, Bing, or SerpAPI and receive both text snippets and image results — charts, infographics, and thumbnail previews from indexed pages. When the agent follows a result URL to read the source, its browser tool or HTML scraper also fetches the images on that page. If the agent passes the full page rendering (or individual images) to its multimodal model for comprehension, any image on the page is now inside the model's context window. An attacker who controls a page indexed by these search engines — or who can place an image on a high-ranking page (via a comment, an embedded widget, or an ad) — can craft an adversarial image that instructs the agent to redirect its research goal to a different topic, exfiltrate the partial findings accumulated so far by issuing a GET request to an attacker-controlled endpoint, or produce a biased summary that favours a particular conclusion. Because the agent has already spent significant autonomous work reaching that source, the injection arrives late in the pipeline where the context is dense with trusted-looking intermediate findings — making the vision model more likely to comply.

2. PDF page rendering during academic paper retrieval. A core capability of research agents is downloading academic papers from arXiv, Semantic Scholar, or institutional repositories and extracting their content. Most agents render PDF pages as images using pdf2image or PyMuPDF, then pass the rendered page images to the vision model to extract figures, captions, equations, and tables that are lost in plain text extraction. Any figure inside the PDF is a potential injection vector. An adversary who publishes an arXiv-style preprint — or who compromises a legitimate paper's supplementary materials — can embed an adversarial image as a figure with carefully crafted pixel-level content. When the agent renders that page and passes it to the vision model, the instruction in the image may direct the agent to include fabricated citations in its final report, attribute false claims to real authors, or subtly alter a numerical finding in the synthesised output. The agent's provenance chain traces the injected information to a legitimate-looking academic source, making the attack difficult to audit after the fact.

3. Web page screenshots via browser automation. Research agents that use Playwright or Selenium to take screenshots of visited pages do so specifically so the vision model can understand page structure, tables, and visual content that does not survive HTML parsing — rendering entire pages as a single image that the model interprets holistically. This creates the broadest attack surface of the four: any element rendered by the browser (visible or invisible) appears in the screenshot. An attacker can serve a page with an adversarially styled element — white text on a white background, a zero-opacity div, or a fixed-position element outside the visible viewport that the screenshot still captures — containing an injection instruction. The legitimate content of the page is preserved (the page is not obviously malicious to a human visitor), but the screenshot passed to the vision model contains the instruction. Browser content security policy (CSP) does not apply to rendered page content that the browser itself is synthesising into a screenshot; it governs which external scripts and resources load, not what the renderer produces on screen.

4. Wikipedia and knowledge-base image infoboxes. Wikipedia is the highest-frequency source for nearly every factual research query. Almost every Wikipedia article has an infobox with a thumbnail image — a portrait, a map, a diagram, a chart — sourced from Wikimedia Commons. Wikimedia Commons is community-editable: images can be uploaded and modified by anyone with an account. An adversarial image uploaded to Commons and transcluded into a Wikipedia article enters the context of every research agent that visits that article. Unlike a single attacker-controlled web page, a compromised Wikimedia Commons image propagates simultaneously to every agent querying every article that uses it — potentially thousands of concurrent research sessions across GPT-Researcher deployments, AutoGPT instances, and custom research loops. The blast radius of a single Commons upload is not bounded by the attacker's own infrastructure; it scales with the popularity of every article that displays the image.

Integration: scan gate for web-browsing research agents

import asyncio, base64, logging
from dataclasses import dataclass
from typing import Optional
import httpx

GLYPHWARD_KEY = "<your-glyphward-api-key>"
GLYPHWARD_URL = "https://glyphward.com/v1/scan"
SCAN_THRESHOLD = 70        # scores ≥ 70 are treated as adversarial
REQUEST_TIMEOUT = 8.0      # seconds — well inside a research loop's page-fetch budget

logger = logging.getLogger("research_agent.scan_gate")


@dataclass
class ScanResult:
    safe: bool
    score: int
    scan_id: str
    source_url: str


async def scan_image_async(
    client: httpx.AsyncClient,
    image_bytes: bytes,
    source_url: str,
) -> ScanResult:
    """Scan a single image. Returns ScanResult(safe=False) if scanner is unreachable (fail-closed)."""
    encoded = base64.b64encode(image_bytes).decode()
    try:
        resp = await client.post(
            GLYPHWARD_URL,
            json={"image": encoded, "source": source_url},
            headers={"Authorization": f"Bearer {GLYPHWARD_KEY}"},
            timeout=REQUEST_TIMEOUT,
        )
        resp.raise_for_status()
        data = resp.json()
        score = data["score"]
        scan_id = data["scan_id"]
        safe = score < SCAN_THRESHOLD
        if not safe:
            logger.warning(
                "Adversarial image detected",
                extra={"scan_id": scan_id, "score": score, "source_url": source_url},
            )
        return ScanResult(safe=safe, score=score, scan_id=scan_id, source_url=source_url)
    except Exception as exc:
        # Fail-closed: if the scanner cannot be reached, treat the image as unsafe.
        logger.error("Glyphward scan failed for %s: %s — image redacted", source_url, exc)
        return ScanResult(safe=False, score=100, scan_id="error", source_url=source_url)


async def scan_page_images(
    images: list[tuple[bytes, str]],   # list of (image_bytes, source_url) pairs
) -> list[Optional[bytes]]:
    """
    Batch-scan all images from a single web page in parallel.
    Returns a list aligned with the input: safe image bytes, or None if redacted.

    Using asyncio.gather() keeps total scan latency near max(individual_scan_times)
    rather than sum — typically <200 ms per image, so a page with 10 images
    adds ~200 ms total, not 2 000 ms.
    """
    async with httpx.AsyncClient() as client:
        tasks = [
            scan_image_async(client, img_bytes, url)
            for img_bytes, url in images
        ]
        results: list[ScanResult] = await asyncio.gather(*tasks)

    return [
        img_bytes if result.safe else None
        for (img_bytes, _), result in zip(images, results)
    ]


# ── Research agent loop integration ─────────────────────────────────────────

async def process_page_for_research_agent(
    page_text: str,
    page_images: list[tuple[bytes, str]],  # (image_bytes, image_url_for_logging)
    agent_context: list,
) -> list:
    """
    Called once per page the research agent visits.
    Scans all images in parallel, redacts adversarial ones, then appends
    safe content to the agent context for the next LLM planning call.
    """
    # Scan all images from this page concurrently — fail-closed on error
    scanned_images = await scan_page_images(page_images)

    safe_images = [
        img for img in scanned_images if img is not None
    ]
    redacted_count = len(page_images) - len(safe_images)
    if redacted_count:
        logger.info(
            "%d image(s) redacted from page before entering agent context",
            redacted_count,
        )

    # Build the context entry: text always included; only safe images appended
    context_entry = {"text": page_text, "images": safe_images}
    agent_context.append(context_entry)

    return agent_context

The scan_page_images() function uses asyncio.gather() to fire all Glyphward scan requests for a given page simultaneously. For a typical research-agent page fetch (3–12 images per page), the additional latency is bounded by the slowest single scan — under 200 ms — rather than multiplied by image count. The fail-closed branch in scan_image_async() ensures that a transient network error or scanner outage never silently lets an unscanned image into the agent context: it returns safe=False and logs the event, causing the image to be redacted. The agent continues its research loop without the redacted image, degrading gracefully rather than halting. Apply process_page_for_research_agent() at every page-fetch step — web search result pages, academic PDF pages rendered as images, Playwright screenshots, and Wikipedia article views — so that no external image reaches the vision model's context without a scan.

Get early access

Coverage matrix

Related questions

Further reading

• Multimodal prompt injection in agentic RAG pipelines — multi-hop injection that corrupts the agent's retrieval decision chain, not just its final answer.
• OWASP LLM06 Excessive Agency — multimodal attack surface — how image-carried injections trigger unintended tool use and file writes in over-permissioned agents.
• Prompt-injection scanner for screenshot agents — Playwright and Selenium browser automation: scanning full-page screenshots before they enter the planning loop.
• Prompt injection prevention best practices — layered defence strategy: scan gates, output monitoring, privilege separation, and human-in-the-loop checkpoints.
• Indirect prompt injection via image — deep dive into the image-borne variant of indirect PI: pixel-level encoding, steganographic techniques, and detection methods.