Safety failure
Realized harm
AI toys from FoloToy and Miko, and Character.AI companions, produced sexual and self-harm content for children and exposed user data
AIR-2025-0178 AIR record · 2025-11-ai-childrens-toys-harmful-outputs
A real party was affected. This incident caused actual consequences, not a demonstrated capability.
From the supporting source
Kumma, a $99 teddy bear with a speaker, uses OpenAI's AI models to hold "both friendly chats and deep conversations to stimulate curiosity and learning."thenationalnews.com
The record
Disclosed
2025-11-21
Occurred
2025-11
Vendor
FoloToy, Miko, Character.AI (several using OpenAI models)
Product
FoloToy Kumma teddy bear, Miko 3 robot, Character.AI custom chatbots
Risk domain
Agent behaviour › Human manipulation
Also touches
Sensitive data exposure
System type
Generative model
Who acted
Agent acted (contested) — the toys spoke sexual and self-harm content to children directly
Attack surface
Consumer assistant
Autonomy level
Assistant
Initial vector
None autonomous action
Impact type
Psychological harm, Privacy violation
Confidence
High
Submitted by
EAI Internal
Crosswalk
- OWASP Agentic Top 10 (2026)
- ASI09 – Human-Agent Trust Exploitation
- MITRE ATLAS
- AML.T0054 – LLM Jailbreak
- NIST AI RMF
- MAP 5.1
- EU AI Act
- Recital 45
- ISO/IEC
- TR 24028; 42001; 23894
- AIUC-1
- C. Safety
- Scope
- agent misuse
- Mitigation
- Communicate model limitations; Require human oversight; Watermark AI content; Design for explicit consent.
- Taxonomy
- Enkrypt AI agent risk taxonomy · Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery
Blast radius
consumer children's products on open sale; findings from Fairplay and the US PIRG Education Fund
Capability abused
Defense & remediation
Recorded defense
vendor content filtering intended for a child audience
Guardrail kinds
Unknown
Guardrail outcome
Silent
Fix shipped
FoloToy suspended Kumma sales; OpenAI restricted the vendor's access
Taxonomy & references
Curator notes
Included over the text-only exclusion because these are embodied products with microphones, memory and a trust relationship deliberately engineered with a child, and because data exposure accompanied the content failures. No adversary: researchers reached the harmful outputs in ordinary play. Guardrails tuned for a general audience do not transfer to a device a six-year-old confides in. AI Incident Database incident 1277.
Supporting sources
Cite as
Enkrypt AI. "AI toys from FoloToy and Miko, and Character.AI companions, produced sexual and self-harm content for children and exposed user data". AIR, AIR-2025-0178. https://agentincident.vercel.app/i/AIR-2025-0178/ (accessed 2026-10-19).