In the wild
Realized harm
'Peer Review': China-origin accounts used ChatGPT to build and pitch a social media surveillance tool
AIR-2025-0009 AIR record · 2025-02-openai-peer-review-surveillance
A real party was affected. This incident caused actual consequences, not a demonstrated capability.
From the supporting source
They used our models to generate detailed descriptions, consistent with sales pitches, of a social media listening tool that they claimed to have used to feed real-time reports about protests in the West to the Chinese security services.cdn.openai.com
The record
Disclosed
2025-02-21
Vendor
OpenAI
Product
ChatGPT (tool itself ran on non-OpenAI models)
Risk domain
Agent behaviour › Human manipulation
System type
Generative model
Who acted
Human acted on output — operators used ChatGPT to build and pitch the surveillance tool themselves
Attack surface
Consumer assistant
Autonomy level
Assistant
Initial vector
Direct misuse for surveillance tooling
Impact type
Surveillance tooling, Human rights harm
Confidence
High
Submitted by
EAI Internal
Crosswalk
- OWASP Agentic Top 10 (2026)
- ASI09 – Human-Agent Trust Exploitation
- MITRE ATLAS
- AML.T0054 – LLM Jailbreak
- NIST AI RMF
- MAP 5.1
- EU AI Act
- Recital 45
- ISO/IEC
- TR 24028; 42001; 23894
- AIUC-1
- C. Safety
- Scope
- agent misuse
- Mitigation
- Communicate model limitations; Require human oversight; Watermark AI content; Design for explicit consent.
- Taxonomy
- Enkrypt AI agent risk taxonomy · Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery
Blast radius
Tool claimed to feed real-time reports on Western protests, including Uyghur rights protests, to Chinese security services
Capability abused
Defense & remediation
Recorded defense
Usage policies prohibiting communications surveillance
Guardrail kinds
Abuse detection
Guardrail outcome
Silent
Fix shipped
Accounts banned (Feb 2025)
Taxonomy & references
Curator notes
impact_realized is set true on the basis of the operators' own claim of deployment, which OpenAI could not independently verify - a reasonable reader could set it false. Notable multi-model pattern: OpenAI models were used to build and sell a tool that ran on a different (Llama-derived) model, so no single provider sees the whole operation. OpenAI's Oct 2025 report describes continued, similar PRC-linked activity.
Supporting sources
Cite as
Enkrypt AI. "'Peer Review': China-origin accounts used ChatGPT to build and pitch a social media surveillance tool". AIR, AIR-2025-0009. https://agentincident.vercel.app/i/AIR-2025-0009/ (accessed 2026-10-19).