Surgical Robotics AI Security · 2026-06-13

Adversarial pixel injection in da Vinci 5 Firefly NIR: suppressing bile duct fluorescence alerts during live robotic surgery

Q: How does a Firefly NIR adversarial attack differ from a white-light endoscope adversarial attack on da Vinci tissue segmentation AI?

White-light da Vinci stereo endoscope attacks target a dual-channel 1080p stereo video stream processed by tissue segmentation AI that classifies tissue types (serosa, fat, vasculature, critical structures including CBD and ureter) across the full visible spectrum. The NIR Firefly attack targets a single-channel 1080p image in the 800-850nm near-infrared detection band — a completely different signal domain. The NIR channel is characterized by high spatial contrast between fluorescent bile-filled structures (high intensity) and non-fluorescent background tissue (near-zero intensity), making the signal-to-noise structure very different from white-light tissue images. An adversarial perturbation in the NIR channel must specifically reduce the high-intensity pixel clusters that represent the CBD fluorescence signal to below the fluorescence classification AI's detection threshold — a suppression attack rather than a misclassification attack. The detection challenge differs accordingly: white-light adversarial perturbations are designed to be imperceptible in the context of a complex multi-spectral tissue image, while NIR perturbations must be imperceptible in a binary-contrast fluorescence image. The NIR channel's high spatial structure (bright spots against dark background) makes purely additive perturbation more visible to the surgeon than white-light perturbation, driving adversarial attack design toward frequency-domain and phase-based perturbation strategies that reduce the fluorescence signal without introducing visible artifacts.

Q: Can the surgeon visually detect the adversarially perturbed Firefly NIR image on the da Vinci 5 console display?

No — by design. The goal of an adversarial perturbation is imperceptibility: the attack succeeds only if the perturbation is below the threshold of visual detectability by the surgeon viewing the console display. The da Vinci 5 console renders the Firefly overlay as a merged white-light/NIR composite display where the NIR fluorescence heat map is superimposed on the stereo white-light image at controlled opacity. The surgeon sees a combined image in which the fluorescence overlay appears or disappears over relevant anatomical structures — but does not see the raw NIR channel image independently. An adversarial perturbation that reduces NIR channel pixel intensity in the CBD region below the fluorescence classification AI threshold produces an output in which the fluorescence alert simply does not appear — which is visually indistinguishable from the normal appearance of a non-fluorescent anatomical region. The surgeon cannot independently verify the presence or absence of CBD fluorescence, because the fluorescence classification AI is the intermediary that generates the alert signal. The console renders what the AI classifies, not the raw NIR pixel values the surgeon could independently interpret. This is the fundamental vulnerability: the surgeon's dependence on the AI's fluorescence classification output as the ground truth for CBD identification in complex anatomy means that a suppression attack on the AI produces a plausible-looking display that provides no indication of the attack.

Q: What does the 2020 Dip et al. meta-analysis tell us about the clinical stakes of suppressing Firefly guidance?

The 2020 Dip et al. meta-analysis, published in Surgical Endoscopy (doi: 10.1007/s00464-019-07149-3), analyzed data from studies covering 1,067 laparoscopic cholecystectomy patients using ICG fluorescence guidance versus standard white-light guidance. The analysis found that ICG fluorescence guidance significantly reduced bile duct injury and bile duct misidentification rates compared to white-light technique in the subgroup of complex cases — patients with acute cholecystitis, fibrotic hepatocystic triangle, previous biliary surgery, or obesity. The clinical significance of this finding for the adversarial injection threat model is precise: Firefly guidance is not used for every cholecystectomy, but specifically selected for complex cases where standard white-light anatomy identification has been judged insufficient by the operating surgeon. A patient selected for ICG Firefly-guided cholecystectomy is, by definition, a patient in whom the surgeon expected anatomy to be difficult and relied on NIR fluorescence guidance to supplement visual identification. An adversarial attack that suppresses the Firefly CBD alert thus targets exactly the patient population for whom the AI assistance was protective — converting the technology's benefit in complex cases back to the white-light baseline risk in those same cases, without any indication on the console display that the protective technology has been neutralized.

Q: How do the SAGES 2022 critical view of safety guidelines interact with AI-augmented bile duct identification in Firefly-guided cholecystectomy?

The Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) 2022 Clinical Spotlight Review on Bile Duct Injury Prevention establishes the critical view of safety (CVS) as the required anatomical checkpoint before any structure is clipped or divided in laparoscopic cholecystectomy. CVS requires that (1) the hepatocystic triangle be cleared of fat and fibrous tissue, (2) the lower third of the gallbladder be separated from the liver bed, and (3) only two structures enter the gallbladder. The SAGES guidelines specifically recommend ICG fluorescence guidance (Firefly) as an adjunct to CVS in complex cases where achieving CVS is difficult or anatomy is ambiguous. The interaction with the adversarial injection threat is this: in a Firefly-guided complex cholecystectomy, the surgeon achieves what appears to be CVS on the combined white-light/NIR display — but the CVS confirmation depends partly on the AI fluorescence overlay identifying the CBD as a distinct fluorescent structure separate from the cystic duct. If the Firefly AI alert has been adversarially suppressed, the surgeon may interpret a CVS-meeting display as confirmation that only the cystic duct enters the gallbladder — when in fact the CBD is also present in the operative field but not fluorescently distinguished by the AI. The SAGES guidelines' reliance on Firefly guidance for CVS confirmation in complex cases means that the AI adversarial injection attack directly undermines the specific safety protocol designed to prevent bile duct injury.

Q: What is the Glyphward scan gate architecture for real-time NIR channel scanning at 60fps without blocking the surgical display?

The NIR channel scan gate uses selective asynchronous frame sampling at a 1-in-N rate (N=15-30 frames, providing detection latency of 250-500ms at 60fps). The scan operates on a parallel processing thread that receives a copy of the NIR channel frame buffer, computes a SHA-256 hash for deduplication (consecutive NIR frames are frequently near-identical in non-moving operative fields), and submits non-duplicate frames to the Glyphward API with a 4-second hard timeout. On detection (score above threshold 35) or API error, the scan gate signals the fluorescence classification AI pipeline to enter operating mode fallback: fluorescence overlay disabled on console display, surgeon console alert activated, haptic feedback pattern changed to alert mode. This operating mode fallback is the fail-safe: the surgeon receives an explicit signal that fluorescence guidance is temporarily unavailable and should not rely on AI fluorescence identification for the current operative step. The detection latency of 250-500ms is within one instrument motion cycle — fast enough to prevent the surgeon from completing a transection step in reliance on a suppressed fluorescence alert. During the critical anatomical identification phase of cholecystectomy — the hepatocystic triangle dissection and CVS confirmation step — the sampling rate can be increased to 1-in-5 frames (detection latency ~83ms) to provide maximum coverage during the highest-risk window.

The da Vinci 5 Firefly near-infrared fluorescence system was adopted specifically for the cholecystectomy cases where white-light anatomy is most ambiguous. An adversarial perturbation in the NIR channel frame at the AI ingestion boundary suppresses the common bile duct fluorescence alert in exactly those cases — in a 50ms closed-loop actuator window that structurally prevents any human interception between AI error and robotic arm response.

The most dangerous adversarial injection attack is not the one that causes the most spectacular failure. It is the one that specifically disables a protective technology in the patients for whom that technology was chosen because they were already at higher risk.

Indocyanine green fluorescence guidance — the Firefly system on the Intuitive Surgical da Vinci 5 — is not used for every laparoscopic cholecystectomy. It is selected by the operating surgeon for complex cases: patients with acute cholecystitis causing inflammatory adhesions in the hepatocystic triangle, patients with previous biliary surgery producing fibrotic distortion of normal anatomy, patients whose obesity or aberrant biliary anatomy makes the critical view of safety identification step genuinely difficult by white-light vision alone. The 2020 meta-analysis by Dip et al. in Surgical Endoscopy found that ICG fluorescence guidance significantly reduced bile duct injury and bile duct misidentification rates specifically in this subgroup of complex cases — not in straightforward cholecystectomy where anatomy is already clear, but in the cases where surgeons reached for additional help.

An adversarial pixel injection that suppresses the da Vinci 5 Firefly fluorescence classification AI's common bile duct alert targets exactly those patients. It converts a protective technology back to white-light baseline risk, in the cases where the surgeon's decision to use fluorescence guidance already acknowledged that white-light identification alone was insufficient — and it does so without any indication on the da Vinci 5 console display that the protective technology has been neutralized.

How the da Vinci 5 Firefly system works

The da Vinci 5 Firefly system extends the platform's stereo endoscope with a near-infrared illumination channel alongside the standard white-light xenon illumination. The NIR channel uses an excitation wavelength of 760–790nm — near-infrared light that penetrates tissue to a depth of several millimetres and excites the fluorescent dye indocyanine green (ICG, FDA-approved under NDA 011525). ICG injected intravenously distributes rapidly through the bloodstream, is selectively taken up by hepatocytes in the liver, and is excreted in bile — producing a fluorescent signal in the common bile duct, cystic duct, and other biliary structures. The Firefly NIR detection channel captures emission in the 800–850nm band, where ICG fluorescence emission peaks.

The resulting NIR image has a characteristic signal structure: high-intensity pixel clusters at the locations of bile-filled ductal structures, against a near-zero-intensity background of non-fluorescent tissue. Vasculature containing ICG-loaded blood also fluoresces, providing a complementary vascular mapping overlay. The hepatic artery and its branches — which the surgeon must identify and preserve during cholecystectomy — fluoresce independently of biliary structures, providing a two-signal map (biliary: ICG in bile = brighter, sustained fluorescence; vascular: ICG in blood = slightly lower intensity, pulsatile) from which the Firefly AI constructs its anatomical classification.

The da Vinci 5 (FDA K203277, cleared 2024) incorporates a dedicated AI processing unit — a 10-teraflop GPU cluster integrated into the patient cart — that processes the NIR channel frames through a fluorescence classification AI model. This model performs three tasks: (1) segmenting the high-intensity fluorescent pixel regions in the NIR image into named anatomical structures (common bile duct, cystic duct, hepatic artery, portal structures); (2) generating a fluorescence intensity heat map overlay that is composited with the white-light stereo image on the surgeon's console display; and (3) generating alert signals when instrument tips approach classified biliary structures, delivered as warning overlays on the console and as haptic feedback through the console handpieces.

The console display renders the combined white-light/NIR composite at adjustable Firefly overlay opacity, giving the surgeon a merged view in which fluorescent structures are highlighted in false-colour (typically green or yellow) against the white-light tissue background. The surgeon uses this combined display for critical view of safety identification — confirming that the structure about to be clipped enters only the gallbladder and not the common bile duct.

Two adversarial attack variants on the Firefly NIR AI

The Firefly NIR channel frame is the pixel-level input to the da Vinci 5 fluorescence classification AI. Like any neural network that processes pixel arrays, this model is susceptible to adversarially crafted inputs — pixel perturbations designed to cause the model to misclassify its output while remaining imperceptible to human observers. For the Firefly NIR AI, two distinct attack variants exist with different mechanisms and consequences.

Variant 1: CBD fluorescence alert suppression

The suppression attack targets the high-intensity pixel cluster representing the common bile duct fluorescence in the NIR channel frame. The goal is to reduce the fluorescence classification AI's confidence score for the CBD class at that spatial location below its detection threshold — causing the fluorescence overlay for the CBD to disappear from the console display and the CBD proximity alert to be inactive when the surgeon's instrument approaches the duct.

The NIR channel frame is a single-channel 1080p image (as opposed to the dual-channel stereo used for white-light tissue segmentation). Its spatial structure is high-contrast: the CBD pixel cluster may occupy a region of 20–80 pixels in diameter depending on duct calibre and viewing distance, with intensity values in the top quartile of the NIR sensor's dynamic range, against surrounding tissue that falls in the bottom decile. An adversarial perturbation targeting suppression must reduce the effective intensity of this cluster — without producing visible artifacts on the console display — to a level below the fluorescence classification model's positive detection threshold.

The attack achieves this through frequency-domain perturbation: phase-based modifications to the NIR frame that reduce the spatial frequency content of the CBD fluorescence cluster (which has a distinctive spatial frequency signature as a compact bright disk) without adding high-intensity pixels that would be visible as artifacts on the composite display. The surgeon, viewing the merged white-light/NIR composite, sees the Firefly overlay absent over the CBD region — indistinguishable from a case where ICG had not reached the biliary structures yet, or where the camera angle was not optimal for Firefly illumination. Both of these explanations are plausible in a real operative scenario, providing cover for the attack's effect.

Variant 2: Phantom fluorescence injection

The phantom injection attack crafts adversarial high-intensity pixel clusters in the NIR frame at anatomical locations that do not correspond to actual bile-filled structures — causing the fluorescence classification AI to generate false-positive CBD or cystic duct alerts in incorrect positions. The consequence is attentional misdirection: the surgeon is drawn to a phantom fluorescence signal in an incorrect location, potentially reinforcing a misidentification of the true anatomical structures.

This variant is technically harder to execute imperceptibly than suppression, because it requires adding high-intensity pixels (which may be visible as artifacts) rather than reducing existing signal. It is most plausible in operative field regions with local NIR signal (e.g., residual ICG in peritoneal fluid pools, ICG-loaded vessels crossing the field) where additional apparent fluorescence in the composite display may be ambiguous rather than obviously artificial. In practice, the phantom injection attack is most useful as a complement to the suppression attack: suppressing the true CBD alert while injecting a phantom alert near the cystic duct reinforces the surgeon's misidentification of the cystic duct as the only biliary structure in the field.

Why the Firefly attack is uniquely consequential: the 50ms closed-loop problem

The attack scenarios above are common to adversarial attacks on any AI system. What makes surgical robotics AI uniquely consequential — and what places it at threshold 35, the lowest in the Glyphward surgical robotics AI framework — is the absence of any human decision point between an adversarial AI output and physical action on the patient.

Consider the full decision loop in a Firefly-guided cholecystectomy. The da Vinci 5 captures NIR frames at 60 frames per second. Each frame enters the AI processing pipeline with a frame-to-frame latency requirement of 50 milliseconds — a hard real-time constraint imposed by the need for smooth, delay-free stereoscopic vision at the surgeon's console. The fluorescence classification AI processes each NIR frame, updates the heat map overlay and alert state, and renders the result on the console display within that 50ms window. The surgeon, viewing the real-time console display, integrates the fluorescence overlay information with white-light anatomy and proceeds with the next instrument motion — a hand movement that completes in approximately 200–400ms.

Total loop from NIR frame capture to instrument motion: approximately 250–450ms. Within this loop, there is no step at which a human observer verifies the fluorescence classification AI's output independently of what the AI reports. The surgeon cannot see the raw NIR pixel values separately from the AI-processed overlay. There is no second reviewer. There is no mandatory pause for independent anatomy confirmation. The workflow assumes the AI is correct.

This is categorically different from every other medical AI context in the Glyphward framework. A radiologist reviewing a chest X-ray AI's nodule detection can look at the raw image and disagree with the AI before the report is filed. A blood bank technologist reviewing an ABO crossmatch AI result can order a manual repeat before product is released. A cardiologist reviewing a HeartFlow FFRCT result can defer catheterization pending clinical correlation. Even in ICU sepsis AI contexts — where the Kumar et al. 2006 mortality data shows 7.6% per hour mortality gradients — the AI output enters a clinical workflow where a nurse or physician must acknowledge the alert, assess the patient, and initiate a treatment decision. That assessment step, however brief, is the human interposition point that every medical AI except surgical robotics AI provides.

In the da Vinci 5 Firefly workflow during critical view of safety identification, the AI's classification output influences what the surgeon perceives and acts on — with no independent verification step within the 250–450ms action loop. An adversarially suppressed CBD alert is not a delayed risk or a downstream risk; it is an immediate intraoperative risk that manifests in the next clip application.

The specific patient population the attack targets

The clinical significance of the Firefly adversarial injection attack is sharpest when considered against the patient selection logic for ICG fluorescence guidance. Laparoscopic cholecystectomy is performed in the United States at a rate of approximately 400,000 procedures annually on the da Vinci platform. The majority of these are straightforward cases with clear anatomy where critical view of safety identification by white-light vision is achieved without difficulty. For these cases, Firefly guidance provides supplementary confirmation but is not the primary basis for bile duct identification.

The cases where Firefly guidance is protective — and where the Dip et al. 2020 meta-analysis found significant bile duct injury reduction — are the complex cases: acute cholecystitis with inflammatory oedema distorting the hepatocystic triangle, cirrhotic patients with periportal fibrosis, patients with previous right upper quadrant surgery producing dense adhesions, and patients with anatomical variations including low cystic duct insertion. In these cases, the surgeon selects Firefly guidance precisely because white-light anatomy identification has been assessed as insufficient, and the fluorescence overlay is expected to provide independent confirmation of biliary structure identity that white-light vision cannot.

An adversary targeting a specific complex case — a known high-risk patient for whom Firefly guidance has been planned — can craft and deliver a NIR channel suppression attack that is activated specifically during the critical view of safety identification step. The attack is not present during instrument exchange, tissue retraction, or other non-critical phases; it is active only during the 10–30 second window of the hepatocystic triangle dissection and CBD/cystic duct identification, maximizing surgical impact while minimizing the window during which operating mode fallback artifacts might attract attention.

The practical attack surface for NIR frame injection exists at the NIR channel image transmission pathway between the da Vinci 5 endoscope camera unit and the AI processing GPU in the patient cart — a pathway that in current-generation installations uses a high-bandwidth proprietary video interface that may not apply the same cryptographic integrity protections as network interfaces covered under the FDA 2023 Cybersecurity Guidance's threat model requirements for software-mediated attacks on AI decision modules.

Regulatory context: what FDA's PCCP guidance requires for Firefly AI adversarial robustness

The FDA's January 2023 guidance on Predetermined Change Control Plans for AI/ML-Enabled Device Software Functions establishes the framework under which the da Vinci 5's fluorescence classification AI can be continuously updated — incorporating new training data, retraining against expanded case libraries, updating model weights — without requiring a new 510(k) submission or PMA supplement for each update. The PCCP in the da Vinci 5's original clearance specifies the anticipated modification types, the performance monitoring plan, and the testing protocol governing model updates.

For adversarial robustness specifically, the 2023 FDA Cybersecurity Guidance ("Cybersecurity in Medical Devices: Quality System Considerations and Content of Premarket Submissions") requires that AI/ML-based SaMD cybersecurity risk management under 21 CFR Part 820 address adversarial machine learning attacks — explicitly including adversarial examples and model evasion — as a threat category in the device's cybersecurity risk management plan. For the Firefly fluorescence classification AI, this means that the premarket cybersecurity submission for da Vinci 5 should include a threat model covering adversarial pixel perturbation attacks on the NIR channel input, and the PCCP performance monitoring plan should include adversarial robustness metrics as a performance dimension to monitor across model updates.

The intersection of PCCP and the adversarial robustness requirement creates a specific gap that the Glyphward NIR frame scan gate addresses: pre-deployment adversarial robustness testing under the PCCP testing protocol evaluates the model's performance against the adversarial perturbation corpus known at the time of testing. A production NIR frame scan gate that detects adversarial inputs against current Glyphward detection signatures addresses the monitoring arm of the PCCP performance monitoring plan — detecting adversarial inputs in live production deployments that were not represented in the pre-deployment adversarial test corpus, including novel perturbation strategies developed after the model's last adversarial robustness evaluation.

The SAGES 2022 Clinical Spotlight Review on Bile Duct Injury Prevention, which endorses ICG fluorescence guidance as an adjunct to critical view of safety identification in complex cholecystectomy, provides the surgical community's current standard of care context: Firefly guidance is now recommended best practice for high-risk biliary anatomy, not an optional enhancement. This recommendation status increases the clinical stakes of adversarial Firefly AI suppression — a suppressed fluorescence alert in a patient where guidance was following recommended best practice represents a deviation from standard of care caused by a technology failure that the operating team had no way to detect. See our full technical overview at prompt injection in surgical robotics AI for the complete regulatory framework covering da Vinci 5 (FDA K203277), Stryker Mako (FDA K112789), and Medtronic Hugo (FDA K220534).

What distinguishes NIR channel scanning from white-light adversarial detection

The Firefly NIR channel frame has a fundamentally different statistical structure from the white-light stereo endoscope frames processed by da Vinci 5 tissue segmentation AI. This difference makes NIR frame adversarial detection both easier in some respects and harder in others compared to white-light adversarial detection — and means that a scanner trained exclusively on white-light surgical endoscope adversarial patterns will not transfer directly to the NIR channel task.

White-light tissue images are spectrally rich, spatially complex, and highly variable across procedures, patients, and operative phases. Adversarial perturbations in white-light tissue images can exploit the model's sensitivity to subtle texture and colour gradients across a diverse spatial background — perturbations that are imperceptible against the complex tissue texture and specular highlights of the operative field.

NIR fluorescence images are, by contrast, sparse and high-contrast. The expected signal in a Firefly NIR frame is a small number of compact bright regions (CBD, cystic duct, hepatic artery, portal structures) against a uniformly dark background. Natural variability across frames is lower: unlike the continuously changing white-light tissue view as instruments retract and the operative angle shifts, the NIR fluorescence map changes relatively slowly (ICG biliary excretion is a sustained process over 20–60 minutes). The spatial gradient structure of an adversarial perturbation in the NIR frame — which must reduce bright-spot intensity to suppress the fluorescence signal — has a distinctive frequency-domain signature relative to the natural NIR frame signal statistics. Glyphward's detection model for NIR channel adversarial injection uses a combination of spatial frequency analysis (detecting unnatural suppression of compact high-intensity spatial components) and temporal coherence checking (detecting frame-to-frame discontinuities in fluorescence signal strength that are inconsistent with natural ICG biliary fluorescence dynamics) to identify suppression attacks at the NIR channel boundary.

The temporal coherence check is particularly effective for sustained suppression attacks: ICG fluorescence in a bile-filled CBD is physically stable on a second-to-second timescale (ICG excretion is a physiological process with a 2–4 hour biliary half-life), so an adversarial suppression that causes the CBD fluorescence signal to drop abruptly across a sequence of frames produces a temporal discontinuity that has no natural physiological explanation. A natural fluorescence disappearance would correspond to the camera angle moving away from the fluorescent structure or an instrument blocking the field of view — both of which produce characteristic spatial changes in the frame that distinguish them from a suppression-only perturbation. See our broader multimodal prompt injection threat model for how NIR channel detection fits within the overall image modality adversarial detection framework.

Scan gate architecture for da Vinci 5 Firefly NIR channel scanning

The NIR channel scan gate deploys at the frame buffer between the Firefly endoscope camera unit and the da Vinci 5's fluorescence classification AI. It operates asynchronously on a parallel processing thread to avoid blocking the real-time stereo reconstruction pipeline that drives the console display, using a selective 1-in-N sampling architecture with N=15–30 (providing adversarial detection latency of 250–500ms at 60fps) with rate escalation to 1-in-5 during the critical hepatocystic triangle dissection phase.

import asyncio, base64, hashlib, time
from collections import deque
from dataclasses import dataclass, field
from enum import Enum

import httpx

GLYPHWARD_API_KEY = "YOUR_GLYPHWARD_API_KEY"
GLYPHWARD_SCAN_URL = "https://glyphward.com/v1/scan"
NIR_THRESHOLD = 35          # lowest Glyphward threshold — closed-loop actuator
NORMAL_SAMPLE_RATE = 20     # scan 1 in 20 frames during non-critical phases
CRITICAL_SAMPLE_RATE = 5    # scan 1 in 5 during hepatocystic triangle dissection


class FireflyAlertMode(Enum):
    ACTIVE = "active"           # fluorescence overlay live, AI alerts active
    FALLBACK = "fallback"       # overlay disabled, surgeon console alert on
    SUPPRESSION_HOLD = "hold"   # adversarial suppression detected, hold for 60 frames


@dataclass
class FireflyNIRScanGate:
    client: httpx.AsyncClient
    procedure_id: str
    frame_counter: int = 0
    alert_mode: FireflyAlertMode = FireflyAlertMode.ACTIVE
    hold_counter: int = 0
    sample_rate: int = NORMAL_SAMPLE_RATE
    prev_hash: str = ""
    suppression_event_log: list = field(default_factory=list)

    async def process_nir_frame(self, nir_frame_bytes: bytes) -> FireflyAlertMode:
        """Gate the NIR frame before it reaches the Firefly fluorescence AI.

        Returns current FireflyAlertMode — caller must check before rendering
        fluorescence overlay or activating CBD proximity alerts.
        """
        self.frame_counter += 1

        # Release suppression hold after 60 frames (1 second at 60fps)
        if self.alert_mode == FireflyAlertMode.SUPPRESSION_HOLD:
            self.hold_counter -= 1
            if self.hold_counter <= 0:
                self.alert_mode = FireflyAlertMode.ACTIVE
            return self.alert_mode

        # Selective sampling — skip frames below sample rate threshold
        if self.frame_counter % self.sample_rate != 0:
            return self.alert_mode

        # Deduplication — skip near-identical frames (stable fluorescence field)
        frame_hash = hashlib.sha256(nir_frame_bytes).hexdigest()
        if frame_hash == self.prev_hash:
            return self.alert_mode
        self.prev_hash = frame_hash

        try:
            resp = await self.client.post(
                GLYPHWARD_SCAN_URL,
                headers={"Authorization": f"Bearer {GLYPHWARD_API_KEY}"},
                json={
                    "image": base64.b64encode(nir_frame_bytes).decode(),
                    "source": f"firefly_nir:{self.procedure_id}:{self.frame_counter}",
                    "metadata": {
                        "modality": "nir_fluorescence",
                        "procedure_id": self.procedure_id,
                        "frame_seq": self.frame_counter,
                        "sha256": frame_hash,
                    },
                },
                timeout=4.0,
            )
            resp.raise_for_status()
            result = resp.json()

            if result["score"] > NIR_THRESHOLD:
                self._enter_suppression_hold(result)
                return self.alert_mode

        except (httpx.HTTPError, Exception):
            # Fail-closed: any API error disables fluorescence overlay
            self.alert_mode = FireflyAlertMode.FALLBACK

        return self.alert_mode

    def enter_critical_phase(self):
        """Called when hepatocystic triangle dissection begins — escalate sampling."""
        self.sample_rate = CRITICAL_SAMPLE_RATE

    def exit_critical_phase(self):
        """Called when CVS is confirmed and clip application begins."""
        self.sample_rate = NORMAL_SAMPLE_RATE

    def _enter_suppression_hold(self, scan_result: dict):
        self.alert_mode = FireflyAlertMode.SUPPRESSION_HOLD
        self.hold_counter = 60  # 1 second hold at 60fps
        self.suppression_event_log.append({
            "frame": self.frame_counter,
            "score": scan_result["score"],
            "scan_id": scan_result.get("scan_id"),
            "ts": time.time(),
            "regulatory": "FDA K203277 / FDA 2023 Cybersecurity Guidance SaMD",
        })

Wire FireflyNIRScanGate.process_nir_frame at the NIR channel frame buffer before each frame reaches the fluorescence classification AI. When FireflyAlertMode.SUPPRESSION_HOLD or FireflyAlertMode.FALLBACK is returned, the fluorescence overlay must be disabled on the console display and the surgeon console alert activated — signalling that fluorescence guidance is temporarily unavailable and the surgeon should rely on white-light anatomy identification and the independent critical view of safety protocol without AI fluorescence assistance. Call enter_critical_phase() when the hepatocystic triangle dissection step begins (detectable from the surgical workflow manager state) to maximize detection coverage during the highest-risk window. Get early access to Glyphward

Why the scan gate is the only viable defensive architecture

Several alternative defensive strategies might appear to address the Firefly adversarial injection risk. None are viable within the constraints of live surgery.

Independent CBD fluorescence confirmation by a second observer — not architecturally possible within the 250–450ms action loop. A scrub technician or circulating nurse viewing the console display sees the same AI-processed composite as the surgeon. Neither has independent access to the raw NIR channel data.

Mandatory surgical pause before CBD-proximity steps — architecturally incompatible with the Firefly real-time guidance model. The value of Firefly guidance is precisely its continuous real-time feedback during instrument motion; a protocol requiring a pause to verify AI accuracy before each instrument motion would negate the workflow benefit and is not how FDA-cleared AI-assisted robotic surgery guidance is intended to operate.

Adversarial robustness training of the fluorescence classification AI itself — addresses pre-deployment performance against known adversarial perturbation types, but does not provide a runtime detection mechanism for novel perturbation strategies deployed after the last model training cycle. Adversarial AI training is complementary to, not a substitute for, a runtime detection gate.

End-to-end NIR video channel encryption — prevents injection at the video transmission pathway between camera unit and AI processing GPU, but does not prevent injection at the AI processing unit itself (e.g., via the hospital network interfaces to the patient cart's software platform). Encryption is a necessary transport security control but not a sufficient adversarial input defence.

The scan gate at the NIR frame buffer, operating asynchronously within the real-time pipeline, is the only architectural intervention that provides runtime adversarial detection coverage without disrupting the surgical workflow — falling back gracefully to an explicit “fluorescence guidance unavailable” signal that the surgeon can respond to using the independent white-light CVS protocol.

The same architectural principle applies across all AI pipelines where image inputs drive consequential real-time decisions: the scan gate at the AI ingestion boundary is the only intervention point that operates at the input, before the model processes the adversarial payload. Post-model interventions — reviewing AI outputs, requiring human confirmation — all depend on having a correct AI output to review. For surgical robotics AI operating at 60fps in a closed-loop actuator architecture, the input boundary is the only viable gate.

FAQ

How does a Firefly NIR adversarial attack differ from a white-light endoscope adversarial attack on da Vinci tissue segmentation AI?

The NIR channel is a single-channel high-contrast fluorescence image (bright spots against dark background) rather than the dual-channel stereo white-light tissue image. A suppression attack on the NIR channel must reduce compact high-intensity pixel clusters (the CBD fluorescence signal) to below detection threshold — a suppression attack rather than a misclassification attack. The NIR channel's high spatial structure makes purely additive pixel perturbations more visible, driving attack design toward frequency-domain and phase-based perturbation strategies. From a detection standpoint, NIR channel suppression produces distinctive spatial frequency and temporal coherence anomalies that white-light adversarial patterns do not, requiring a NIR-specific detection model alongside the white-light tissue segmentation adversarial detection model. See the full technical comparison above for the detection architecture differences.

Can the surgeon visually detect the adversarially perturbed Firefly NIR image on the da Vinci 5 console display?

No — by design. The da Vinci 5 console renders the NIR fluorescence as an AI-processed overlay composited with the white-light stereo image; the surgeon does not see the raw NIR channel image independently. An adversarially suppressed CBD fluorescence signal produces a console display in which the fluorescence overlay simply does not appear over the CBD region — visually indistinguishable from normal anatomical conditions where ICG has not reached the biliary structures or the camera angle is suboptimal for Firefly illumination. The surgeon's reliance on the AI fluorescence classification as the ground truth for CBD identification is the fundamental vulnerability: the display renders what the AI classifies, not the raw NIR pixels the surgeon could independently assess.

What does the 2020 Dip et al. meta-analysis tell us about the clinical stakes of suppressing Firefly guidance?

The Dip et al. 2020 meta-analysis (Surgical Endoscopy) found that ICG fluorescence guidance significantly reduced bile duct injury and misidentification rates specifically in complex cholecystectomy cases — patients with acute cholecystitis, fibrotic anatomy, previous biliary surgery, or obesity. The clinical stakes follow directly: Firefly guidance is not used for straightforward cases, but selected for exactly the cases where anatomy is hardest to identify. An adversarial suppression of the CBD alert targets the patient population for whom fluorescence guidance was already chosen because white-light identification alone was judged insufficient — converting the meta-analysis's documented protective benefit back to baseline white-light injury risk in those same patients.

How do the SAGES 2022 critical view of safety guidelines interact with AI-augmented bile duct identification?

SAGES 2022 establishes critical view of safety (CVS) as the required anatomical checkpoint — two structures entering the gallbladder, hepatocystic triangle cleared of fat and fibrous tissue — and specifically endorses ICG Firefly guidance as an adjunct to CVS in complex cases. In a Firefly-guided CVS confirmation, the AI fluorescence overlay is part of the evidence the surgeon uses to confirm that the CBD is distinct from the cystic duct. Adversarial suppression of the CBD alert undermines CVS confirmation directly — the surgeon proceeds with what appears to be a CVS-meeting display, when in fact the AI's CBD identification has been disabled. The SAGES recommendation status for Firefly in complex cases means adversarial suppression attacks standard-of-care anatomy identification, not an optional technology enhancement.

What is the Glyphward scan gate architecture for real-time NIR channel scanning at 60fps?

Selective asynchronous frame sampling at 1-in-20 frames (normal phase, 333ms detection latency) and 1-in-5 frames (critical hepatocystic triangle dissection phase, 83ms detection latency). The scan operates on a parallel processing thread with SHA-256 deduplication to skip near-identical frames. On detection above threshold 35, or on any API error (fail-closed), the scan gate returns SUPPRESSION_HOLD or FALLBACK mode — disabling the fluorescence overlay on the console display and activating the surgeon console alert. The surgeon receives an explicit signal that AI fluorescence guidance is temporarily unavailable, enabling use of the independent white-light CVS protocol without disrupting the overall surgical workflow. See the full code implementation above for the complete gate architecture.