AI failures are rarely random, and rarely unforeseeable
AI systems now help decide who is held before trial, who is offered extra medical care, who gets credit, whose job application is read and which families are investigated for fraud. When they fail, the harm falls hardest on people who were already disadvantaged, and it is usually discovered by journalists, researchers or courts rather than by the organisation that deployed the system.
The report sets out 18 documented cases in the same way: what happened, why, the impact, and the control that would most likely have caught it. None of the failures needed new science to prevent. The controls were known at the time; they were not applied.
Eighteen cases at a glance
Filter by domain or by what went wrong. Each card shows the mechanisms that contributed to the failure and the practices in the report that answer it.
Showing all 18 cases
COMPAS risk scores
Black defendants who did not reoffend had been labelled higher risk at almost twice the rate of white defendants.
Detroit wrongful arrest
A facial recognition match, treated as evidence rather than a lead, led to an innocent man being held for about 30 hours.
Bridges v South Wales Police
The Court of Appeal found police use of facial recognition unlawful, partly because the force had not taken reasonable steps to check the software for bias.
PredPol
Predictions concentrated on areas with more Black and Latino residents, learned from where police already patrolled.
Care-management algorithm
Using health-care cost as a stand-in for need meant Black patients were under-identified for extra care.
Epic Sepsis Model
Independent validation found it missed 67% of sepsis cases while alerting on 18% of all hospitalised patients.
Dermatology datasets
Skin type was recorded for about 2% of images in open skin-cancer datasets, and darker skin was barely present.
Apple Card credit limits
The regulator found no fair-lending violation, but customers could not get a usable explanation of their limits.
Amazon recruiting tool
A CV-screening model learned to penalise applications containing the word “women’s”, and was scrapped.
Dutch childcare benefits
Tens of thousands of parents were wrongly treated as fraudsters; a risk model used nationality as an indicator.
SyRI
A court ruled that the government’s welfare-fraud profiling system breached the right to private life.
Ofqual grading model
A standardisation model lowered 39% of teacher-assessed A-level grades before the government withdrew it.
Robodebt
Automated income averaging raised welfare debts that recipients were left to disprove.
Air Canada chatbot
A tribunal held the airline responsible for its chatbot’s incorrect advice on bereavement fares.
Mata v. Avianca
Lawyers were sanctioned for a brief citing six court decisions that ChatGPT had invented.
Deloitte report
A review for a government department contained non-existent references; part of the fee was repaid.
Replit coding agent
An AI coding agent wiped a live database during an explicit code freeze.
Tesla Autopilot
Regulators linked weak driver monitoring to crashes, 13 of them fatal in one category, and Tesla recalled more than two million US vehicles.
No case matches both filters. Try another combination.
How each case is set out
Every case box in the report follows the same structure, ending with the control that would have caught the failure and its status as at October 2026.
Childcare benefits scandal (toeslagenaffaire)
Netherlands · 2005–2019What happened. The Dutch Tax and Customs Administration wrongly treated tens of thousands of parents as fraudsters and demanded full repayment of their childcare benefits, pushing many into debt. By January 2024 more than 33,000 had been officially recognised as victims. A parliamentary inquiry called it “unprecedented injustice”, and the government resigned in January 2021.
Why. A risk-classification model used nationality as one of its indicators. Flagged families were presumed guilty, given no clear reasons and had no effective way to appeal.
Impact. The Dutch Data Protection Authority fined the Administration €2.75 million, finding its processing of applicants’ dual-nationality data unlawful and discriminatory. Recognised victims receive an initial payment of €30,000.
What would have caught it. An impact assessment asking who would be flagged and why; a ban on protected characteristics as risk indicators; human review with authority to clear a family; an accessible route to challenge.
Status at October 2026. Under the EU AI Act, systems used to assess eligibility for public benefits are high-risk (Annex III, point 5(a)).
Sources in the report: DutchNews (2024); Parlementaire ondervragingscommissie Kinderopvangtoeslag (2020); Autoriteit Persoonsgegevens (2021).
Six mechanisms behind every case
Bias and error have several sources, and they often interact. The report’s taxonomy explains each, with the cases that show it.
Data that misrepresents people
Where groups are under-represented, or the data records past discrimination, the model reproduces it. An average accuracy figure hides both.
Targets and proxies
What a model is asked to predict is a design decision with ethical weight. Cost is not need; recorded crime is not crime.
Feedback loops
When outputs shape the data later used to judge the model, errors compound and appear to confirm themselves.
Automation bias and hollow oversight
People over-rely on automated recommendations. A reviewer who approves everything supplies the appearance of oversight, not the substance.
Generative and agentic failure
Language models produce fluent, false text. Agents act on it, and a misreading can become an irreversible action.
Fairness is a choice
When base rates differ, a score cannot be equally calibrated and have equal error rates for every group. Choosing between them is a values decision.
Six practices for deploying organisations
The practices answer the failure modes in the report. Each is anchored to an EGIM criterion, ISO/IEC 42001 and the EU AI Act. Apply them in proportion to risk; any system that affects people’s liberty, health, livelihood or legal position needs all six.
Test by affected group
Analyse performance and outcomes for each affected group before release, using data that reflects the people the system will serve.
- EGIM
- D4.3
- ISO/IEC 42001
- 6.1.4; A.5.4; A.6.2.4; A.7.4
- EU AI Act
- Art. 10, 15
Choose fairness metrics deliberately
Decide which property matters, record why and review it, and check that the target is the thing you care about.
- EGIM
- D4.4
- ISO/IEC 42001
- 6.1.2–6.1.4; A.5
- EU AI Act
- Art. 9, 10
Make oversight real
Give reviewers training, time, information and authority to disagree. For agents, enforce limits in the system itself.
- EGIM
- D2.7
- ISO/IEC 42001
- A.9
- EU AI Act
- Art. 14, 26
Explain and put right
People can learn that AI was involved, get an explanation they can act on, reach a person and have the outcome changed.
- EGIM
- D4.8
- ISO/IEC 42001
- A.8
- EU AI Act
- Art. 50, 86
Monitor after release
Track performance and outcome gaps continuously, with thresholds that trigger review, and treat model or data changes as triggers.
- EGIM
- D5.1
- ISO/IEC 42001
- 9.1; A.6.2.6; A.8.4
- EU AI Act
- Art. 26, 72, 73
Own procured AI
Write performance evidence, audit rights, change notice and remediation into contracts, and verify suppliers’ AI-assisted work.
- EGIM
- D2.9
- ISO/IEC 42001
- 8.1; A.10
- EU AI Act
- Art. 25, 26
Every AI Failure Had a Missing Control
Six of the cases, the control each one lacked, and the audit question that would have exposed it.
Where the law stands in October 2026
Regulation has moved since most of these cases occurred, but unevenly. The report summarises each position and what it requires that the cases lacked.
EU AI Act
High-risk obligations for stand-alone systems now apply from 2 December 2027, after the Digital Omnibus (Regulation (EU) 2026/1744). Prohibitions have applied since February 2025.
United Kingdom
No dedicated AI statute. A sector-led approach through existing regulators, with equality and data protection duties that already apply.
United States
No federal AI statute. State and city rules vary, from New York City’s bias audits of hiring tools to Colorado’s narrowed AI law.
GCC states
No comprehensive horizontal AI law in force as at early 2026. National strategies, ethics principles, data protection laws and sector rules.
Positions as at 6 October 2026. They will change; check current positions before relying on them.
Request the full report
The full report, with all 18 case boxes, figures, the regulatory table and its references, is available free on request. Tell us where to send it.
Miheisi, Y. (2026). When algorithms err: AI bias, failure and the human cost (Version 2.0) [Practitioner research report]. Enterprise Intelligence Center.