Safety failure
Realized harm
Waymo vehicles illegally passed stopped school buses at least 19 times, including five after a software fix Waymo said had resolved it
AIR-2025-0187 AIR record · 2025-12-waymo-school-bus-passing-recall
A real party was affected. This incident caused actual consequences, not a demonstrated capability.
From the supporting source
In a November 20 letter posted by NHTSA, the Austin Independent School District said five incidents occurred in November after Waymo said it had made software updates to resolve the issue and asked the company to halt operations around schobgr.com
The record
Disclosed
2025-12-04
Occurred
2025-09
Vendor
Waymo (Alphabet)
Product
Waymo Driver
Risk domain
Agent behaviour › Unsafe actuation
System type
Classical autonomy (no generative model)
Who acted
Agent acted — the vehicles illegally passed stopped school buses 19 times
Attack surface
Autonomous operations
Autonomy level
Fully autonomous
Initial vector
None autonomous action
Impact type
Physical safety
Confidence
High
Submitted by
EAI Internal
Crosswalk
- OWASP Agentic Top 10 (2026)
- ASI01 – Agent Goal Hijack
- MITRE ATLAS
- AML.T0048 – External Harms
- NIST AI RMF
- MEASURE 2.6; MANAGE 1.3
- EU AI Act
- Annex III §1(c)
- ISO/IEC
- TR 24028; 24029-1; 23894
- AIUC-1
- C. Safety
- Scope
- agent misuse
- Mitigation
- Require confirmation for destructive actions; Implement dry-run modes; Use layered safety controls.
- Taxonomy
- Enkrypt AI agent risk taxonomy · Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery
Blast radius
at least 19 recorded incidents across the 2025 school year in Texas and Georgia; children boarding and alighting school buses
Capability abused
Defense & remediation
Recorded defense
school-bus detection logic, updated mid-incident
Guardrail kinds
Unknown
Guardrail outcome
Silent
Fix shipped
voluntary NHTSA software recall filed December 2025 after the first fix failed
Taxonomy & references
Curator notes
The important fact is not the 19 violations but the five that happened after the fix. A vendor asserting a behavioural defect is resolved is a claim about a learned policy's behaviour in an open world, and here it did not hold — the school district had to keep counting. Pairs with the 2023 Tesla school-bus record as the same scenario at two different autonomy levels. AI Incident Database incident 1300.
Supporting sources
Cite as
Enkrypt AI. "Waymo vehicles illegally passed stopped school buses at least 19 times, including five after a software fix Waymo said had resolved it". AIR, AIR-2025-0187. https://agentincident.vercel.app/i/AIR-2025-0187/ (accessed 2026-10-19).