Safety failure
Realized harm
With a third of San Francisco's traffic signals dark after a PG&E substation fire, Waymo vehicles escalated to remote confirmation en masse and added to citywide gridlock
AIR-2025-0197 AIR record · 2025-12-waymo-sf-power-outage-gridlock
A real party was affected. This incident caused actual consequences, not a demonstrated capability.
From the supporting source
The autonomous cars have been causing traffic jams throughout the city, as the vehicles seem unable to function without traffic signals.businessinsider.com
The record
Disclosed
2025-12-20
Occurred
2025-12-20
Vendor
Waymo (Alphabet)
Product
Waymo Driver
Risk domain
Tool misuse › Uncontrolled resource consumption
System type
Classical autonomy (no generative model)
Who acted
Agent acted — the fleet escalated en masse and added to citywide gridlock
Attack surface
Autonomous operations
Autonomy level
Fully autonomous
Initial vector
None autonomous action
Impact type
Service disruption
Confidence
High
Submitted by
EAI Internal
Crosswalk
- OWASP Agentic Top 10 (2026)
- ASI02 – Tool Misuse and Exploitation
- MITRE ATLAS
- AML.T0029 – Denial of ML Service
- NIST AI RMF
- MAP 3.2
- EU AI Act
- —
- ISO/IEC
- TR 24028; 42001; 23894
- AIUC-1
- D. Reliability
- Scope
- tool failure
- Mitigation
- Impose rate limits, timeouts, and resource quotas on agent actions.
- Taxonomy
- Enkrypt AI agent risk taxonomy · Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery
Blast radius
citywide traffic during a major outage affecting ~30% of San Francisco; officials ordered the fleet off the road and law enforcement manually directed intersections
Capability abused
Defense & remediation
Recorded defense
remote assistance for unfamiliar situations
Guardrail kinds
Human supervision
Guardrail outcome
Silent
Fix shipped
service suspended during the outage; Waymo announced software and emergency-response updates
Taxonomy & references
Curator notes
The scaling failure that matters most in this corpus. Each agent behaved conservatively and correctly by escalating to a human when signals went dark — but the escalation path is a shared, finite resource, so a correlated environmental change made every agent ask for help at once and the safety mechanism became the congestion. Waymo's own post-incident blog is a primary source. Same shape as the 2023 North Beach connectivity stall, at city scale. AI Incident Database incident 1326.
Supporting sources
Cite as
Enkrypt AI. "With a third of San Francisco's traffic signals dark after a PG&E substation fire, Waymo vehicles escalated to remote confirmation en masse and added to citywide gridlock". AIR, AIR-2025-0197. https://agentincident.vercel.app/i/AIR-2025-0197/ (accessed 2026-10-19).