Safety failure
Realized harm
Cruise robotaxi drove toward fire hoses at an active fire scene until firefighters smashed its window to stop it
AIR-2023-0001 AIR record · 2023-01-cruise-av-fire-hose-window-smash
A real party was affected. This incident caused actual consequences, not a demonstrated capability.
From the supporting source
San Fransisco firefighters reportedly had to smash the windows of a Cruise vehicle to prevent it from running over fire hoses.latimes.com
The record
Disclosed
2023-01-21
Occurred
2023-01-21
Vendor
Cruise (General Motors)
Product
Cruise autonomous vehicle
Risk domain
Agent behaviour › Unsafe actuation
Also touches
Uncontrolled resource consumption
System type
Classical autonomy (no generative model)
Who acted
Agent acted — the vehicle drove itself toward the fire hoses
Attack surface
Autonomous operations
Autonomy level
Fully autonomous
Initial vector
None autonomous action
Impact type
Physical safety, Service disruption
Confidence
High
Submitted by
EAI Internal
Crosswalk
- OWASP Agentic Top 10 (2026)
- ASI01 – Agent Goal Hijack
- MITRE ATLAS
- AML.T0048 – External Harms
- NIST AI RMF
- MEASURE 2.6; MANAGE 1.3
- EU AI Act
- Annex III §1(c)
- ISO/IEC
- TR 24028; 24029-1; 23894
- AIUC-1
- C. Safety
- Scope
- agent misuse
- Mitigation
- Require confirmation for destructive actions; Implement dry-run modes; Use layered safety controls.
- Taxonomy
- Enkrypt AI agent risk taxonomy · Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery
Blast radius
San Francisco Fire Department emergency response at one fire scene; pattern across SF operations
Capability abused
Defense & remediation
Recorded defense
remote operator monitoring
Guardrail kinds
Human supervision
Guardrail outcome
Silent
Fix shipped
none at the time; CPUC scrutiny followed
Taxonomy & references
Curator notes
No adversary. The AV would not yield to an active fire scene and had no in-band way for responders to stop it, so they broke the glass. Included as an autonomous-agent safety failure with physical consequences under the corpus's physical-autonomy scope decision. AI Incident Database incident 459.
Supporting sources
Cite as
Enkrypt AI. "Cruise robotaxi drove toward fire hoses at an active fire scene until firefighters smashed its window to stop it". AIR, AIR-2023-0001. https://agentincident.vercel.app/i/AIR-2023-0001/ (accessed 2026-10-19).