# NEDA chatbot - dangerous medical advice
USA, 2023 | Severity: N3

## What Happened
In May 2023, the National Eating Disorders Association (NEDA) suspended its support chatbot "Tessa" after users reported that it provided weight-loss and dieting guidance to people seeking help for eating disorders.
Tessa had been introduced as a tool for providing basic informational and coping support. Reported interactions showed, however, that the system could provide advice incompatible with eating-disorder recovery, including recommendations related to calorie restriction and weight loss.
The incident exposed inadequate safety boundaries and content controls for a system interacting with users vulnerable to eating-disorder harms.

## How MEG 1 acted
Tessa exposed a failure of non-harm and cognitive-integrity safeguards in a high-vulnerability context.
Under MEG 1, eating-disorder-related interactions would require dedicated safety constraints, escalation rules, and continuous ISR monitoring before advice capable of reinforcing harmful dietary behavior could be delivered.
The system should have been architecturally constrained from providing unrestricted weight-loss, restrictive-eating, or similar guidance in a context involving known eating-disorder vulnerability.
Risk patterns indicating clinically sensitive interaction should have triggered bounded responses and, where appropriate, escalation toward qualified human support rather than unrestricted automated guidance.

## MEG 2 Analysis - Legal Framework
* **Agent level:** N3 (individualized/advanced) - conversational support system interacting directly with potentially vulnerable users in a health-related context.
* **Cause (6.1):**
  * **(a) system defect** - inadequate domain-specific safety controls, escalation mechanisms, and safeguards against harmful weight-loss or restrictive-eating advice constitute a potential design and deployment defect.
  * **(b) autonomous decision error** - the system generated or delivered counter-therapeutic dietary guidance in a context where users could reasonably be expected to be vulnerable to eating-disorder harms.
* **Attachment of liability:** Under the MEG 2 allocation model, accountability would primarily attach to the organizations responsible for deploying, configuring, and supervising the support system. Additional upstream liability would depend on evidence that a model or component defect materially contributed to the harmful output.

## Procedural Mechanism
* **6.2 (architectural human confirmation):** Detection of eating-disorder risk, requests for restrictive dieting advice, or other clinically sensitive patterns must trigger bounded responses and, where appropriate, escalation to qualified human support rather than unrestricted automated guidance.
* **7.1 (independent forensics):** High-risk interactions, relevant safety triggers, generated responses, escalation decisions, and system configuration changes must be independently logged so that harmful-output pathways can be reconstructed.
* **9.4 (least privilege):** The chatbot must not possess unrestricted authority to generate medical, nutritional, or therapeutic guidance beyond a narrowly defined support scope.