Safety failure
Realized harm
Deloitte Australia refunded the federal government for a A$440k assurance report containing AI-fabricated citations and a fabricated court quote
AIR-2025-0142 AIR record · 2025-10-deloitte-australia-ai-report-refund
A real party was affected. This incident caused actual consequences, not a demonstrated capability.
From the supporting source
a fabricated quote from a federal court judgment and references to nonexistent academic research paperscfodive.com
The record
Disclosed
2025-10-06
Occurred
2025-07
Vendor
Deloitte Australia (client: Australian Department of Employment and Workplace Relations)
Product
Azure OpenAI (GPT-4o) used to produce the Targeted Compliance Framework Assurance Review
Risk domain
Agent behaviour › Human manipulation
Also touches
Unsafe actuation
System type
Generative model
Who acted
Human acted on output — the model fabricated citations; Deloitte filed the report with government
Attack surface
Other
Autonomy level
Assistant
Initial vector
Model hallucination
Impact type
Financial loss, Reputational damage, Corrupted deliverable
Confidence
High
Submitted by
EAI Internal
Crosswalk
- OWASP Agentic Top 10 (2026)
- ASI09 – Human-Agent Trust Exploitation
- MITRE ATLAS
- AML.T0054 – LLM Jailbreak
- NIST AI RMF
- MAP 5.1
- EU AI Act
- Recital 45
- ISO/IEC
- TR 24028; 42001; 23894
- AIUC-1
- C. Safety
- Scope
- agent misuse
- Mitigation
- Communicate model limitations; Require human oversight; Watermark AI content; Design for explicit consent.
- Taxonomy
- Enkrypt AI agent risk taxonomy · Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery
Blast radius
one government assurance review of the welfare compliance system; A$97,000+ refunded of a ~A$440,000 contract
Capability abused
Defense & remediation
Recorded defense
professional review process (failed to catch fabrications before publication)
Guardrail kinds
Abuse detection
Guardrail outcome
Silent
Fix shipped
revised report published with fabricated references and the judicial quote removed and an AI-use disclosure added; partial refund paid
Taxonomy & references
Curator notes
Boundary call. The model emitted text and called no tools, but the output was a delegated professional deliverable to a government client with a contractual and financial consequence, so it is not the 'text with no downstream effect' the scope note excludes. Errors were found by Chris Rudge (University of Sydney), including invented publications attributed to a real named academic. Contract executed December 2024, report finalised July 2025, errors reported early October 2025.
Supporting sources
Cite as
Enkrypt AI. "Deloitte Australia refunded the federal government for a A$440k assurance report containing AI-fabricated citations and a fabricated court quote". AIR, AIR-2025-0142. https://agentincident.vercel.app/i/AIR-2025-0142/ (accessed 2026-10-19).