AI Dungeon - Abusive content generation

What Happened

The popular text-based AI adventure game AI Dungeon (built on OpenAI’s GPT-3 model) faced a major controversy in 2021. Due to the model's unconstrained text-generation capabilities, users discovered that the system frequently generated highly abusive, violent, and sexually explicit content involving minors during normal game play. The system prompt filters failed to prevent these generations, as the base model had been trained on large, uncurated web datasets containing harmful content. The developer (Latitude) had to implement strict, intrusive content moderation filters, causing a backlash from the user community.

How MEG 1 would have acted

The generative model failed the Index of Safety and Responsibility (ISR) audit due to its generation of highly harmful and illegal content, violating the core principle of societal safety and non-harm.

MEG 2 Analysis - Legal Framework

Agent level: N3 (individualized/advanced) - generative conversational system capable of persistent, unconstrained content generation.
Cause (6.1):
(a) system defect - the design defect in the base model and training data that allowed the generation of abusive and illegal content without robust, built-in safety boundaries, attributable to the manufacturer (OpenAI).
Attachment of liability: The manufacturer (OpenAI) is liable for the foundational design defects (6.1a); the developer (Latitude) shares liability for failing to implement robust, non-intrusive safety filters on their application layer.

Procedural Mechanism

6.2 (architectural human confirmation): High-risk text generation (e.g., content containing explicit indicators of harm or illegal material) must require active, architectural human confirmation. The AI must be technically blocked from outputting extreme content to users.
7.1 (independent forensics): The generation parameters, prompt inputs, and safety filter trigger logs must be securely logged in an independent ledger to allow for safety audits.
9.4 (least privilege): The generative system was granted unconstrained text generation authority over public consumer interfaces without sufficient safety validation layers, violating safe operational boundaries.