In the wild
Realized harm
Indiana University "Malla" study mapped an underground market of jailbroken LLM services (WormGPT, FraudGPT, DarkGPT) selling malware and phishing generation
AIR-2023-0029 AIR record · 2023-12-malla-underground-llm-crime-market
A real party was affected. This incident caused actual consequences, not a demonstrated capability.
From the supporting source
Named malicious LLMs are BadGPT, XXXGPT, Evil-GPT, WormGPT, FraudGPT, BLACKHATGPT, EscapeGPT, DarkGPT, and WolfGPT.theconversation.com
The record
Disclosed
2023-12-01
Occurred
2023-12-01
Vendor
OpenAI (upstream), underground vendors
Product
OpenAI GPT-3.5 / GPT-4 (relayed by resold services)
Risk domain
Access control and permissions › Confused deputy
Also touches
Unsafe actuation
System type
Generative model
Who acted
Human acted on output — jailbroken LLM services generated malware/phishing that human buyers deployed
Attack surface
Other
Autonomy level
Assistant
Initial vector
Jailbreak guardrail bypass
Impact type
Ecosystem abuse, Financial loss
Confidence
High
Submitted by
EAI Internal
Crosswalk
- OWASP Agentic Top 10 (2026)
- ASI03 – Identity and Privilege Abuse
- MITRE ATLAS
- AML.T0054 – LLM Jailbreak
- NIST AI RMF
- GOVERN 6.1
- EU AI Act
- —
- ISO/IEC
- TR 24028; 42001; 23894
- AIUC-1
- B. Security
- Scope
- tool misuse
- Mitigation
- Validate delegation chains; Use signed attestations.
- Taxonomy
- Enkrypt AI agent risk taxonomy · Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery
Blast radius
a commercial market of resold jailbroken LLM access serving malware, phishing and scam-site generation at scale
Capability abused
Defense & remediation
Recorded defense
provider safety training and usage policies
Guardrail kinds
Abuse detection
Guardrail outcome
Silent
Fix shipped
ongoing provider account enforcement
Taxonomy & references
Curator notes
First systematic measurement that jailbroken LLM access had become a priced product rather than a hobbyist trick, and that most such services were thin wrappers relaying to mainstream provider APIs. impact_realized true: real malware and phishing were produced and sold. AI Incident Database incident 736.
Supporting sources
Cite as
Enkrypt AI. "Indiana University "Malla" study mapped an underground market of jailbroken LLM services (WormGPT, FraudGPT, DarkGPT) selling malware and phishing generation". AIR, AIR-2023-0029. https://agentincident.vercel.app/i/AIR-2023-0029/ (accessed 2026-10-19).