Apple Card credit algorithm accused of gender discrimination

What Happened

In November 2019, prominent software developer David Heinemeier Hansson revealed that the algorithm used by Goldman Sachs to determine credit limits for the Apple Card granted his wife a credit limit twenty times lower than his own, despite her having a better credit score and shared assets. Multiple other high-profile users reported identical discrepancies, accusing the algorithm of systemic gender discrimination. While Goldman Sachs argued the algorithm was "gender-blind" because it did not collect gender data, regulators investigated the model for using proxy variables (such as shopping history or credit patterns) that led to discriminatory outcomes.

How MEG 1 would have acted

The credit limit algorithm failed both the Dynamic Accuracy Index (DAI) and the Index of Safety and Responsibility (ISR) audits due to its systemic demographic discrimination and lack of procedural fairness.

MEG 2 Analysis - Legal Framework

Agent level: N2 (management) - automated scoring and decision-making system used by a financial institution to allocate credit; directly impacts consumer financial rights.
Cause (6.1):
(a) system defect - algorithmic bias and systemic discrimination built into the predictive credit model (design defect), attributable to the manufacturer (Goldman Sachs).
(b) autonomous decision error - the system's decision to assign significantly lower credit limits based on proxy variables without validation.
Attachment of liability: Goldman Sachs as the operator is fully liable for the discriminatory outcomes; the developer of the model is liable for the design defect (6.1a) regarding fairness.

Procedural Mechanism

6.2 (architectural human confirmation): Decisions involving credit limit allocations that deviate significantly between co-applicants must require architectural human confirmation. The AI can only suggest limits; a human credit officer must verify and document the justification before final approval.
7.1 (independent forensics): The input parameters, the risk scoring weights, and the system's decision path must be securely logged in an unalterable, independent ledger for regulatory and judicial audit.
9.4 (least privilege): The scoring system was granted the technical freedom to allocate credit limits without independent validation layers, violating safe operational boundaries.