New York City Hall's MyCity chatbot offers illegal advice

What Happened

In March 2024, New York City Hall's "MyCity" AI chatbot—designed to assist small business owners with administrative policies—was found to be generating illegal and dangerous advice. The chatbot (running on Microsoft’s Azure OpenAI services) informed users that employers could legally dock their employees' tips, that landlords could deny housing based on a tenant's source of income, and that businesses could go cashless, all of which directly violated New York City and State laws. Despite these documented hallucinations, City Hall kept the chatbot online, merely adding a disclaimer stating that the tool was "not a substitute for professional legal advice," raising severe concerns about the liability of public authorities providing automated legal misinformation.

How MEG 1 would have acted

The MyCity chatbot failed the Dynamic Accuracy Index (DAI) audit by presenting fabricated, illegal legal advice as official city policy, directly violating the cognitive integrity and informational self-determination of citizens and business owners.

MEG 2 Analysis - Legal Framework

Agent level: N2 (management) - customer interaction chatbot authorized to represent municipal policies and interact directly with small business owners.
Cause (6.1):
(a) system defect (hallucination) - the model's design defect that allowed it to generate false, illegal legal advice (hallucination), attributable to the manufacturer.
(b) autonomous decision error - the chatbot's decision to promise legal permissions contrary to actual municipal and state laws.
Attachment of liability: The operator (NYC City Hall) is fully liable for the representations made by its agent; the disclaimer defense is the classic "AI Decided" defense, which is explicitly invalid under MEG 2.

Procedural Mechanism

6.2 (architectural human confirmation): Commitments involving regulatory compliance, legal advice, or administrative permissions must require architectural human confirmation. The chatbot can only display static, verified policy links; it must not autonomously formulate or interpret legal regulations.
7.1 (independent forensics): Chat logs must be securely stored in an independent ledger, preventing either party from altering the record of the advice given.
9.4 (least privilege): The chatbot was granted the technical freedom to discuss and interpret complex municipal laws without real-time database validation, violating safe operational boundaries.