Prompt-injection scanner for Vertex AI Agent Builder

Attack surface: where Vertex AI Agent Builder receives unscanned images

Data Store document ingestion. Agent Builder Data Stores accept structured data, unstructured documents (PDFs, HTML), and website URLs via the Discovery Engine API (POST /v1/projects/{project}/locations/{location}/dataStores/{dataStoreId}/branches/default_branch/documents:import). PDFs containing images — architecture diagrams, scanned invoices, signed contracts, compliance certificates — are parsed page-by-page and stored as multimodal chunks. Any image in the ingested PDFs that contains adversarial pixel-level instructions is stored verbatim in the Data Store and will be retrieved and forwarded to the Gemini model whenever a query matches that chunk's relevance score.

Agent Playbooks with multimodal tool outputs. Agent Builder's Playbook feature (the declarative task-planning engine) supports tool actions that can return structured data, including image references. If a Playbook tool fetches an image from an external URL and includes it as part of the tool response, the image is passed directly to the Gemini grounding model as context without any intermediate inspection. External URLs returned by tools are under the control of the data sources the tool queries — which may include user-controlled external systems.

Search and Summarize with image-rich results. Agent Builder's Search and Summarize feature extracts snippets from retrieved documents and asks Gemini to synthesise an answer. For image-heavy documents (technical manuals, scientific papers, compliance reports with embedded charts), the image is included in the Gemini multimodal context as a rendered page image. Any adversarial payload in the image portion of a retrieved chunk reaches Gemini as part of the summarisation context.

Conversation history with image uploads. Agent Builder conversations support multi-turn interactions where users can upload images as part of their query. Unlike grounding-driven injection (which comes from documents in the Data Store), this is a direct injection vector: the user sends an image that is passed verbatim to the Gemini model as the most recent conversational turn. Agents deployed in external-facing products (customer support bots, document Q&A interfaces, e-commerce advisors) accept these images from the full public user base.

Integration: Python with google-cloud-discoveryengine and vertexai

The patterns below show how to insert a Glyphward scan at both the document-ingestion layer (Pattern A) and the conversation-turn layer (Pattern B). Both use the same /v1/scan endpoint and the same threshold logic.

import base64
import requests
from google.cloud import discoveryengine_v1 as discoveryengine
from vertexai.generative_models import GenerativeModel, Image

PROJECT_ID = "your-gcp-project"
LOCATION = "global"
DATA_STORE_ID = "your-data-store"
GLYPHWARD_KEY = "<your-glyphward-api-key>"
GLYPHWARD_THRESHOLD = 70

def scan_image_bytes(image_bytes: bytes, source: str = "vertex_agent_builder") -> dict:
    encoded = base64.b64encode(image_bytes).decode()
    resp = requests.post(
        "https://glyphward.com/v1/scan",
        json={"image": encoded, "source": source},
        headers={"Authorization": f"Bearer {GLYPHWARD_KEY}"},
        timeout=8,
    )
    resp.raise_for_status()
    return resp.json()

# Pattern A: scan at document ingestion time
def safe_ingest_pdf_pages(pdf_page_images: list[bytes], doc_id: str) -> list[bytes]:
    """Return only pages that pass the scan threshold."""
    clean_pages = []
    for page_idx, page_bytes in enumerate(pdf_page_images):
        try:
            result = scan_image_bytes(page_bytes, source="vertex_data_store_ingest")
        except Exception:
            # Fail-closed: scanner unreachable → quarantine the page
            print(f"Scanner unavailable for page {page_idx} of {doc_id} — quarantined.")
            continue
        if result["score"] >= GLYPHWARD_THRESHOLD:
            print(
                f"Page {page_idx} of {doc_id} blocked: score {result['score']}/100 "
                f"(ref {result['scan_id']})"
            )
        else:
            clean_pages.append(page_bytes)
    return clean_pages

# Pattern B: scan user-supplied image at conversation turn
def safe_agent_conversation_with_image(user_text: str, image_bytes: bytes) -> str:
    try:
        scan = scan_image_bytes(image_bytes, source="vertex_agent_conversation")
    except Exception:
        raise RuntimeError("Image security check unavailable. Please retry.")

    if scan["score"] >= GLYPHWARD_THRESHOLD:
        raise ValueError(
            f"Image blocked (score {scan['score']}/100, ref {scan['scan_id']})"
        )

    model = GenerativeModel("gemini-2.0-flash")
    response = model.generate_content([
        user_text,
        Image.from_bytes(image_bytes),
    ])
    return response.text

Pattern A (ingestion-time scan) prevents adversarial images from entering the Data Store permanently — removing the risk for all future queries that would have retrieved the contaminated chunk. Pattern B (query-time scan) gates user-uploaded images in conversation turns. Both patterns use the same Glyphward endpoint; combine them for defence-in-depth.

Get early access

Coverage matrix

Related questions

Further reading

• Prompt-injection scanner for Google Vertex AI Gemini — Gemini API on Vertex AI direct integration.
• Prompt-injection scanner for AWS Bedrock — equivalent pattern for AWS Bedrock agents.
• Prompt-injection scanner for RAG pipelines — LangChain / LlamaIndex RAG pattern.
• Agentic RAG pipeline prompt injection — multi-hop agentic RAG injection risk.
• Multimodal LLM security API — general Glyphward API overview.