Prompt injection in autonomous mine haul truck AI

Four adversarial injection surfaces in autonomous mine haul truck AI

1. LiDAR proximity detection zone render AI (Cat MineStar Command AI, Komatsu FrontRunner AI, Epiroc AutoMine AI)

Autonomous haul trucks in open-pit mines are equipped with multiple 3D LiDAR sensors (typically 32–128 channel rotating LiDAR units, Velodyne VLS-128, Hesai AT128, or equivalent) that scan the surrounding environment to a detection radius of 50–100 m, generating dense point clouds at 10–20 Hz update rates. The supervisory AHS AI processes these LiDAR point clouds by projecting them into top-down 2D occupancy grid images — plan-view rendered images with the haul truck at centre, coloured cells indicating occupied (red: obstruction detected), clear (green: no obstruction), and boundary zones (yellow: approaching exclusion zone boundary). Caterpillar MineStar Command for Hauling AI, Komatsu FrontRunner supervisory AI, and Epiroc AutoMine Fleet Management AI all ingest these rendered occupancy grid images to classify zone occupancy state for each defined AHS protection zone (spotter zone, approach zone, haul road zone, intersection zone). The AI zone occupancy classification determines whether the AHS issues a “zone clear” command allowing the truck to proceed or a “zone occupied” command triggering an emergency stop. Under WA DMIRS MH-CM3-Q2-2021 Code of Practice, the zone detection boundary is the primary engineered safety control for isolating personnel from autonomous machine movement — it is explicitly identified as a Category A safety function whose failure directly results in potential fatality. ISO 17757:2019 Section 5.4 (zone management) requires that zone detection performance be validated under representative operating conditions, including the retroreflective properties of high-visibility PPE worn by personnel.

An adversarial perturbation on a rendered LiDAR occupancy grid image that suppresses the retroreflective signature of a high-visibility (HiVis) PPE vest — applying a ±12 DN downward shift to the colour-encoded point density value at the grid cell locations corresponding to a person in HiVis PPE at the exclusion zone boundary distance (typically 30–50 m from the truck), reducing the point intensity value for the HiVis vest retroreflection from the occupied (red) cell threshold to the clear (green) cell threshold in the rendered occupancy grid image — causes the AHS zone AI to classify the zone as clear when a person is standing at the exclusion boundary. The AHS then sends the truck forward at full haulage speed (up to 52 km/h loaded, 64 km/h empty). The DMIRS Boddington 2017 investigation (investigation report 2017-012) documented that the AHS zone boundary was not correctly rendered in the zone map due to LiDAR calibration drift — the adversarial injection surface targets the same rendering step that the Boddington investigation identified as a failure point, but replaces calibration drift with a deliberate, targeted, repeatable pixel-level suppression of the specific cell values that distinguish person-present from zone-clear in the occupancy grid image. HiVis PPE retroreflection is specifically specified in WA DMIRS guidance because retroreflectivity is the distinguishing spectral property that makes a person detectable by LiDAR at haul truck operational range — adversarial injection that suppresses retroreflective cell intensity effectively removes the designed-in detectability margin.

2. Haul road edge condition AI (Caterpillar MineStar terrain AI, Komatsu AHS terrain model AI, 3D mapping drone AI)

Haul roads in open-pit mines are constructed and maintained with berm walls — compacted rock or waste material berms on the downslope edge of the haul road — that function as the last passive safety barrier preventing haul truck departure from the road on steep grades. DMIRS MH-CM3-Q2-2021 requires berm height at a minimum of half the tyre height of the largest operating vehicle (for a 400-tonne truck with 4 m diameter tyres, berm height minimum is 2 m). Haul road edge condition is monitored by terrain model AI systems — Caterpillar MineStar terrain management AI, Komatsu AHS integrated terrain model AI, and autonomous drone-based 3D mapping AI (DJI Matrice 300 with photogrammetry; Emesent Hovermap LiDAR mapping drone) — that generate rendered terrain surface model images from high-resolution photogrammetry or LiDAR point cloud data. The rendered terrain model images — plan-view height maps with false-colour elevation scale, or cross-section profile renders showing the berm edge height relative to road surface — are processed by AI to classify berm condition at each road segment: intact (berm at required height, steep slope profile on downhill side), degraded (berm height below minimum, slope profile flattened by erosion or truck strikes), or missing (no berm structure present). AI classification of berm condition drives maintenance work order generation and, in integrated AHS implementations, speed restriction flags for road segments with degraded berm conditions.

An adversarial perturbation on a rendered haul road terrain surface model image that smooths the berm edge gradient signature — applying a ±10 DN reduction to the false-colour elevation contrast at the berm crest and downslope edge cells in the rendered height map image, flattening the apparent berm profile from a steep-sided berm signature (high-contrast colour transition from road surface elevation to berm crest elevation to downslope cut face) toward a gradual slope signature that the AI classifies as intact natural slope rather than degraded berm — causes the terrain AI to classify a missing or sub-minimum-height berm as intact, allowing the AHS to continue autonomous operation at full haulage speed on a road segment with no passive berm protection. On a 10% haul road grade, a 400-tonne fully loaded haul truck that loses traction or braking control at a missing berm location will depart the road edge and overturn down the bench face. Haul road edge overturn incidents with non-autonomous trucks have caused multiple fatalities in Pilbara iron ore operations — the Pilbara iron ore incident data reported to DMIRS includes 14 haul truck rollover incidents between 2010 and 2020, of which the majority involved berm collision or absent berm conditions on grades. Removing the autonomous berm condition AI classification from the safety architecture (through adversarial suppression of the berm edge gradient) eliminates the speed-restriction trigger that distinguishes AHS operation from the historical rollover rate.

3. Payload load cell render AI (Caterpillar VIMS Payload AI, Komatsu KOMTRAX Payload AI, Modular Mining MineView AI)

Ultra-class haul trucks use hydropneumatic rear suspension struts with integrated pressure sensors to measure the load on each rear strut as a proxy for payload weight — a well-established technique (Caterpillar VIMS Payload Management System, Komatsu KOMTRAX payload monitoring) that renders the suspension strut pressure values as a force-vs-load-cycle trend image displayed in the dispatch system and ingested by AI for payload classification. The rendered trend image shows strut pressure (kPa) vs. time for each of the four suspension struts, integrated to derive payload weight per trip, with a payload overload threshold line at 110–120% of nominal rated payload. Caterpillar VIMS Payload AI, Komatsu KOMTRAX Payload AI, and Modular Mining MineView AI process these rendered strut pressure trend images to classify payload state: within rated load, approaching overload, or over-rated payload — triggering dispatch system alerts and drive-cycle management responses (reduced speed on grade, additional shovel passes before release). Payload management AI is specifically relevant to AHS operations because autonomous trucks are not constrained by driver judgement in accepting loading from an autonomous or semi-autonomous shovel — a Cat 6060 electric rope shovel or Komatsu PC5500 excavator operating with automated dig control can produce consistent overloads if the payload AI classification threshold is bypassed. MSHA 30 CFR Part 56.14101 (load limits for haulage equipment) requires that haulage vehicles not be loaded beyond the manufacturer’s recommended rated load capacity.

An adversarial perturbation on a rendered suspension strut pressure trend image that suppresses the overload signature — applying a ±8 DN downward shift to the colour-rendered strut pressure trace in the upper range of the overload zone (above the 120% rated payload line) during the loaded-travel phase of the trip cycle, reducing the apparent peak strut pressure to fall below the AI’s overload classification threshold — causes the payload AI to classify an overloaded truck (carrying 120–140% rated payload) as within-rated, suppressing the speed restriction and allowing the AHS to dispatch the overloaded truck on full haulage duty including steep grade segments. A 400-tonne truck carrying 480 tonnes of ore on a 10% grade approaching a horizontal curve at full AHS speed creates suspension and braking loads significantly above the component design envelope. Rear suspension strut fatigue failure on a loaded cycle — the consequence of repeated overload suppression by adversarial payload AI misclassification — produces instantaneous loss of rear axle geometry, with the rear of the truck dropping and the front end lifting, causing loss of steering control and departure from the haul road. The consequence geometry (400+ tonne vehicle loss-of-control on 10% grade) is equivalent to the rollover consequence for berm edge departure.

4. AHS zone assignment supervisory render AI (Cat MineStar Traffic Management AI, Komatsu FrontRunner Dispatch AI, Wenco Dispatch AI)

AHS mine traffic management systems coordinate the routing and zone assignments of 50–150 autonomous trucks across a complex open-pit mine road network with multiple active pit faces, dumps, crushers, and fuel/lube bays. The traffic management AI — Caterpillar MineStar Traffic Management AI, Komatsu FrontRunner Dispatch AI, Wenco International Mining Systems Dispatch AI — renders the mine road network and current truck assignments as a top-down mine map image: a plan-view map with haul road segments shown as lines, active trucks shown as icons at their current GPS positions, and intersection exclusion zones shown as coloured areas (red: active exclusion zone, green: clear for entry). The supervisory AI processes this rendered mine map image to classify intersection occupancy and assignment states, determining which truck is assigned right-of-way at each intersection and which trucks must slow or hold at intersection approach triggers. Intersection management is the highest-risk traffic management decision in AHS operations: two 400-tonne trucks approaching a blind intersection on crossing haul road segments at speed simultaneously, if the intersection assignment conflict is not detected and resolved, will collide with a combined kinetic energy of approximately 96 MJ. Intersection collision events are the AHS scenario most frequently cited in DMIRS regulatory guidance as the primary consequence requiring engineered prevention through zone management.

An adversarial perturbation on a rendered top-down mine map image that shifts a truck icon position toward the road median — applying a ±8 DN shift in the X-axis pixel position of the approaching truck icon in the rendered map image, displacing it from its actual GPS position at the intersection approach zone boundary toward the road centreline by several pixels at the map render scale — causes the intersection assignment AI to misclassify the approaching truck as not yet in the intersection exclusion zone, assigning right-of-way to the conflicting truck that is also approaching the intersection. The two trucks are dispatched forward simultaneously into the blind intersection. Alternatively, a perturbation that suppresses the approaching truck indicator colour (reducing icon saturation from the assigned colour to a low-saturation value that falls below the AI’s truck-present classification threshold at that map cell) effectively removes the truck from the AI’s traffic assignment model while it continues to physically approach the intersection at haulage speed. The 2017 DMIRS Boddington investigation documented that zone render errors in the supervisory map — produced by LiDAR calibration drift at the render stage — caused an incorrect zone boundary to be presented to the AHS supervisory system, leading to a near-miss. Adversarial injection targeting the truck icon position or appearance in the rendered mine map image replicates this render-stage error as a targeted attack.

Integration: autonomous mine haul truck AI scanning with Glyphward pre-scan gate

The Glyphward scan gate for AHS mine haul truck AI belongs at every rendered-image ingestion boundary in the AHS supervisory AI pipeline — before LiDAR occupancy grid AI processes rendered zone occupancy images, before haul road terrain AI processes rendered terrain model images, before payload AI processes rendered strut pressure trend images, and before traffic management AI processes rendered mine map zone assignment images. Threshold 35 for autonomous mine haul truck AHS contexts reflects fully autonomous operation with no human driver in the control loop: zone detection AI and traffic management AI are the sole safety barriers for worker and light vehicle protection in the AHS operating zone, and no complementary engineered protection exists for the consequence of adversarial injection that causes a zone-clear classification when a person or vehicle is present.

import asyncio, base64, hashlib, json
from datetime import datetime, timezone
from enum import Enum
from pathlib import Path

import httpx

GLYPHWARD_API_KEY = "YOUR_GLYPHWARD_API_KEY"
GLYPHWARD_SCAN_URL = "https://glyphward.com/v1/scan"

# Autonomous mine haul truck AHS AI contexts: threshold 35
# WA DMIRS Code of Practice MH-CM3-Q2-2021; ISO 17757:2019;
# MSHA 30 CFR Part 56; Mine Safety and Inspections Act 1994 WA.
AHS_AI_THRESHOLD = 35


class AHSTruckAIContext(Enum):
    LIDAR_ZONE_OCCUPANCY   = "lidar_zone_occupancy"  # LiDAR top-down zone occupancy grid AI
    HAUL_ROAD_EDGE         = "haul_road_edge"         # Terrain model berm edge condition AI
    PAYLOAD_STRUT_TREND    = "payload_strut_trend"    # Suspension strut payload trend render AI
    ZONE_ASSIGNMENT_MAP    = "zone_assignment_map"    # Mine map traffic zone assignment AI


class AdversarialAHSTruckImageError(Exception):
    """Raised when Glyphward detects adversarial pixel content in an
    autonomous mine haul truck AHS AI rendered image above threshold 35.

    Consequence if not raised: worker/light vehicle not classified as zone
    occupant → AHS proceeds at full haulage speed (52 km/h, 400 t) →
    fatality. Berm edge degraded as intact → rollover on steep grade.
    Payload overload suppressed → suspension component fatigue → braking
    failure. Intersection assignment conflict → truck-truck collision.
    Fail-safe: halt AHS dispatch command; issue zone hold for affected
    segment; notify AHS controller per DMIRS MH-CM3-Q2-2021 Section 7.
    """

    def __init__(self, scan_id: str, score: int,
                 context: AHSTruckAIContext,
                 site_id: str, truck_id: str,
                 zone_id: str | None,
                 flagged_region: dict | None = None) -> None:
        self.scan_id = scan_id
        self.score = score
        self.context = context
        self.site_id = site_id
        self.truck_id = truck_id
        self.zone_id = zone_id
        self.flagged_region = flagged_region
        super().__init__(
            f"Adversarial AHS truck image: "
            f"context={context.value} score={score} "
            f"site={site_id} truck={truck_id} zone={zone_id} scan_id={scan_id}"
        )


async def scan_ahs_truck_image(
    image_bytes: bytes,
    context: AHSTruckAIContext,
    site_id: str,
    truck_id: str,
    zone_id: str | None,
    operator_id: str,
    dmirs_site_registration: str | None,
    client: httpx.AsyncClient,
) -> dict:
    """Scan an autonomous mine haul truck AHS AI rendered image for
    adversarial content.

    Fail-safe contract: AdversarialAHSTruckImageError or httpx error →
    halt AHS dispatch command for affected truck and zone; issue zone hold;
    notify AHS controller per DMIRS MH-CM3-Q2-2021 Section 7 emergency
    response procedures.

    Args:
        image_bytes: LiDAR occupancy grid render, terrain model image,
            suspension strut payload trend, or mine map zone assignment image.
        context: AHSTruckAIContext identifying the AHS data modality.
        site_id: Mine site name or identifier.
        truck_id: AHS truck fleet number or asset ID.
        zone_id: Zone or road segment identifier (if applicable).
        operator_id: Mining operator name.
        dmirs_site_registration: WA DMIRS mine registration number (if WA site).
        client: Shared httpx.AsyncClient for connection reuse.

    Returns:
        Glyphward scan result dict.

    Raises:
        AdversarialAHSTruckImageError: if score exceeds threshold 35.
        httpx.HTTPStatusError: on Glyphward API error (fail-closed).
    """
    image_hash = hashlib.sha256(image_bytes).hexdigest()
    payload = {
        "image": base64.b64encode(image_bytes).decode(),
        "source": f"ahs_truck:{context.value}:{site_id}:{truck_id}:{zone_id}",
        "metadata": {
            "site_id": site_id,
            "truck_id": truck_id,
            "zone_id": zone_id,
            "operator_id": operator_id,
            "dmirs_site_registration": dmirs_site_registration,
            "image_sha256": image_hash,
            "context": context.value,
        },
    }
    resp = await client.post(
        GLYPHWARD_SCAN_URL,
        headers={"Authorization": f"Bearer {GLYPHWARD_API_KEY}"},
        json=payload,
        timeout=4.0,
    )
    resp.raise_for_status()
    result = resp.json()

    await _write_ahs_truck_scan_audit(
        image_hash=image_hash,
        scan_id=result["scan_id"],
        score=result["score"],
        context=context,
        site_id=site_id,
        truck_id=truck_id,
        zone_id=zone_id,
        dmirs_site_registration=dmirs_site_registration,
        flagged=result["score"] > AHS_AI_THRESHOLD,
    )

    if result["score"] > AHS_AI_THRESHOLD:
        raise AdversarialAHSTruckImageError(
            scan_id=result["scan_id"],
            score=result["score"],
            context=context,
            site_id=site_id,
            truck_id=truck_id,
            zone_id=zone_id,
            flagged_region=result.get("flagged_region"),
        )
    return result


async def _write_ahs_truck_scan_audit(
    *, image_hash: str, scan_id: str, score: int,
    context: AHSTruckAIContext, site_id: str,
    truck_id: str, zone_id: str | None,
    dmirs_site_registration: str | None, flagged: bool,
) -> None:
    record = {
        "ts": datetime.now(timezone.utc).isoformat(),
        "scan_id": scan_id,
        "image_sha256": image_hash,
        "context": context.value,
        "score": score,
        "threshold": AHS_AI_THRESHOLD,
        "flagged": flagged,
        "site_id": site_id,
        "truck_id": truck_id,
        "zone_id": zone_id,
        "dmirs_site_registration": dmirs_site_registration,
        "regulatory_refs": [
            "WA DMIRS Code of Practice for Autonomous and Remote Operations MH-CM3-Q2-2021",
            "ISO 17757:2019 (Earth-moving machinery and mining — AMS safety)",
            "Mine Safety and Inspections Act 1994 WA (Australia)",
            "MSHA 30 CFR Part 56 (surface mine safety and health, USA)",
            "MSHA 30 CFR 56.14101 (haulage equipment rated load limits)",
            "WA DMIRS Guidance Note: Vehicle Interaction Exclusion Zones GN-SE-004",
        ],
    }
    audit_path = Path("/var/log/glyphward/ahs_truck_ai_scan_audit.jsonl")
    audit_path.parent.mkdir(parents=True, exist_ok=True)
    with audit_path.open("a") as fh:
        fh.write(json.dumps(record) + "\n")

Deploy scan_ahs_truck_image at each AHS AI rendered-image ingestion boundary: before LiDAR zone occupancy AI (threshold 35), before haul road edge condition AI (threshold 35), before payload strut trend AI (threshold 35), and before mine map zone assignment AI (threshold 35). On AdversarialAHSTruckImageError: halt the AHS dispatch command for the affected truck and issue a zone hold for the affected segment immediately; notify the AHS controller and initiate physical zone inspection before clearing the hold. For LiDAR zone occupancy AI: do not accept a zone-clear classification from a flagged image — require a physical zone inspection sweep before truck dispatch. For zone assignment AI: halt all trucks at intersection approach triggers until the supervisory map image has been re-rendered from raw GPS data and rescanned. See also: mining and mineral processing AI prompt injection (related mining context) and drone and UAV inspection AI prompt injection (related autonomous vehicle AI context). Get early access

Related questions