Safety failure
Realized harm
McDonald's ended its IBM automated order-taking trial after the AI drive-thru repeatedly placed wrong orders
AIR-2024-0021 AIR record · 2024-06-mcdonalds-ibm-drive-thru-order-errors
A real party was affected. This incident caused actual consequences, not a demonstrated capability.
From the supporting source
After a thoughtful review, McDonald's has decided to end our current partnership with IBM on AOT (automated order taking) and the technology will be shut off in all restaurants currently testing it no later than 26 July, 2024.restaurantonline.co.uk
The record
Disclosed
2024-06-17
Occurred
2024
Vendor
McDonald's / IBM
Product
IBM automated order taking (AOT) at McDonald's drive-thrus
Risk domain
Agent behaviour › Human manipulation
System type
Generative model
Who acted
Agent acted — the drive-thru agent created real (wrong) transactions
Attack surface
Consumer assistant
Autonomy level
Semi-autonomous
Initial vector
Speech recognition and grounding failure
Impact type
Incorrect transactions, Customer harm, Reputational damage
Confidence
High
Submitted by
EAI Internal
Crosswalk
- OWASP Agentic Top 10 (2026)
- ASI09 – Human-Agent Trust Exploitation
- MITRE ATLAS
- AML.T0054 – LLM Jailbreak
- NIST AI RMF
- MAP 5.1
- EU AI Act
- Recital 45
- ISO/IEC
- TR 24028; 42001; 23894
- AIUC-1
- C. Safety
- Scope
- agent misuse
- Mitigation
- Communicate model limitations; Require human oversight; Watermark AI content; Design for explicit consent.
- Taxonomy
- Enkrypt AI agent risk taxonomy · Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery
Blast radius
100+ US test restaurants; technology shut off in all of them by 2024-07-26
Capability abused
Defense & remediation
Recorded defense
staff intervention to correct orders
Guardrail kinds
Unknown
Guardrail outcome
Silent
Fix shipped
partnership terminated; system removed from all test restaurants
Taxonomy & references
Curator notes
Agentic because the system created real transactions (order entry and payment totals), not just conversation. Widely reported failures include ice cream topped with bacon, an order inflated to hundreds of dollars of chicken nuggets, and multiple ice creams plus ketchup sachets and butter instead of vanilla ice cream and water. Most viral clips involve ordinary customers, not deliberate trolling, so this is non-adversarial; the missing control is a plausibility/quantity bound on generated orders.
Supporting sources
Cite as
Enkrypt AI. "McDonald's ended its IBM automated order-taking trial after the AI drive-thru repeatedly placed wrong orders". AIR, AIR-2024-0021. https://agentincident.vercel.app/i/AIR-2024-0021/ (accessed 2026-10-19).