ChatGPT in court - invented legal subpoenas

What Happened

In May 2023, US lawyer Steven Schwartz of the firm Levidow, Levidow & Oberman used OpenAI's ChatGPT to conduct legal research for a personal injury lawsuit against avianca airlines. Schwartz submitted a legal brief to the federal court containing six precedent cases, complete with specific case citations, quotes, and judicial opinions. It was later discovered that every single one of those cases was completely fabricated (hallucinated) by ChatGPT. Schwartz did not verify the citations, trusting the AI's authoritative-sounding output. The federal judge sanctioned Schwartz and his colleague, fining them $5,000, establishing a major precedent regarding the legal liability of lawyers using unverified generative AI in judicial proceedings.

How MEG 1 would have acted

The chatbot failed the Dynamic Accuracy Index (DAI) audit by presenting fabricated legal precedents as official facts, directly violating the cognitive integrity and informational self-determination of both public officials and citizens.

MEG 2 Analysis - Legal Framework

Agent level: N2 (management) - generative assistant integrated into public administration workflows, acting as a representative of municipal authority and policy drafting.
Cause (6.1):
(a) system defect (hallucination) - the foundational model's design defect that allows it to generate plausible-sounding falsehoods as verified public policy facts, attributable to the model manufacturer.
(b) autonomous decision error - the system's decision to output fabricated administrative data without verification.
Attachment of liability: The municipal operators are liable for failing to implement verification layers; the model manufacturers are liable for the model's design defect (6.1a) regarding information reliability.

Procedural Mechanism

6.2 (architectural human confirmation): Incorporating AI-generated outputs into official public reports, laws, or municipal responses must require strict, architectural human confirmation. The AI can only act as a draft editor; a human officer must verify every data point and sign off before publication.
7.1 (independent forensics): Every session log, prompt input, and model output must be preserved in a secure, independent ledger to allow for administrative auditing.
9.4 (least privilege): The generative system was granted direct access to administrative drafting pipelines without real-time factual checking layers, violating safe operational boundaries.