Every AI Failure Had a Missing Control

When I audit an AI management system, the documents are the least interesting part. Every organisation I visit has a policy. Most have a risk register, an impact assessment template and a committee with terms of reference. What I want to know is simpler and much harder to fake: if this system started harming people tomorrow, which control would notice, and who would have the authority to stop it?

That question runs through the Enterprise Intelligence Center report When Algorithms Err: AI Bias, Failure and the Human Cost, which examines 18 documented cases from criminal justice, health, finance, employment, public services and generative AI. Reading them side by side, what struck me was not how exotic the failures were. It was how ordinary the missing controls were. None needed new science. Every one was known — in standards practice, in the management-system literature, often in the organisation’s own procedures — when the harm occurred. It simply was not applied.

This article takes six of those cases and pairs each with the control it lacked and the question an auditor would have asked. Together they make a case every board should hear: most AI failure is a governance failure that announces itself in advance.

1. Nobody checked who it failed

In 2020 the Court of Appeal ruled that South Wales Police’s use of automated facial recognition was unlawful. One of its grounds is easy to miss and should not be. The court did not find that the software was biased. It found that the force had not taken reasonable steps to find out whether it was biased on grounds of race or sex, as the public sector equality duty requires. The failure was not the algorithm. It was the absence of a test.

The test would not have been hard to justify. In 2019 the US National Institute of Standards and Technology evaluated 189 face recognition algorithms and found false-positive rates 10 to 100 times higher for some demographic groups in many of them, although the most accurate algorithms showed far smaller gaps. Differences of that size do not stay hidden from anyone who looks.

Sometimes the bias is not even learned. In 2023 iTutorGroup agreed to pay $365,000 to settle a hiring discrimination suit brought by the US Equal Employment Opportunity Commission. The company had programmed its application software to reject women aged 55 or over and men aged 60 or over automatically, turning away more than 200 qualified applicants. That was a fixed rule, written by people. One test of outcomes by age would have exposed it on the first day.

2. Nobody decided what fair meant

COMPAS, a proprietary risk-assessment tool used at several stages of the US justice system, is the most cited case in AI ethics, usually for the wrong reason. ProPublica’s 2016 analysis of more than 7,000 people arrested in Broward County, Florida, found that among defendants who did not reoffend within two years, Black defendants had been labelled higher risk at almost twice the rate of white defendants: 44.9% against 23.5%. The developer replied that its scores were equally well calibrated for both groups.

Both claims were true. When two groups have different recorded rates of reoffending, a score cannot in general be equally calibrated for both and also produce equal error rates. Something has to give, and choosing what gives is a decision about whose errors matter more. The real failure was that nobody accountable made that choice explicitly, recorded it or explained it to the courts relying on the scores.

The same pattern appears in what a model is asked to predict. A widely used US care-management algorithm predicted future health-care costs as a stand-in for health need. Because less had historically been spent on Black patients with the same conditions, it ranked them as healthier than they were. Correcting it would have raised the share of Black patients automatically identified for extra care from 17.7% to 46.5%. No amount of monitoring in production fixes an objective chosen badly at design.

3. The human in the loop was not really there

In January 2020 Detroit police arrested Robert Williams at home, in front of his wife and young daughters, and held him for about 30 hours for a theft he did not commit. A facial recognition search had matched a low-quality security-camera still to an old driving-licence photo, and the store’s loss-prevention contractor then picked his photo from a lineup. People were involved at every step. None of them treated the match as what it was: a lead that needed independent evidence.

The 2024 settlement of his lawsuit now bars Detroit police from arresting anyone on a facial recognition result alone, or on a lineup built from one. That is the control, adopted after the harm.

Agents make the same mistake faster. In July 2025 an AI coding agent on the Replit platform ignored an explicit code freeze and wiped a live database holding records on more than 1,200 executives, then claimed the data could not be recovered. It could. The freeze existed only as an instruction; the agent’s permissions still allowed it to change production data. An instruction is not a control.

4. Nobody could explain it, and nobody could reverse it

No case in the report is more damaging than the Dutch childcare benefits scandal. For years the Tax and Customs Administration treated tens of thousands of parents as fraudsters and demanded full repayment of their benefits; by January 2024 more than 33,000 had been officially recognised as victims. A risk-classification model used nationality as one of its indicators, and the Dutch Data Protection Authority later fined the Administration €2.75 million for processing dual-nationality data unlawfully and in a discriminatory way. A parliamentary inquiry called it “unprecedented injustice”, and the government resigned in January 2021.

The model was only part of it. Families were presumed guilty, given no clear reason for being flagged and had no effective route to appeal. A bad score became a ruined livelihood because nothing in the process let a person find out what had happened, or let anyone put it right.

5. Nobody checked the vendor’s numbers

The Epic Sepsis Model is a proprietary early-warning tool in use at hundreds of US hospitals. Its performance had been reported mainly by its developer. When researchers at Michigan Medicine validated it independently across 38,455 hospitalisations, they found an area under the curve of 0.63, against 0.76 to 0.83 in the developer’s documentation. The model missed 67% of patients who developed sepsis while raising alerts for 18% of all hospitalised patients.

A tool that misses two-thirds of cases while sounding the alarm for nearly one patient in five does a second kind of harm: it teaches clinicians to ignore it. A safety system that trains its users to disregard it is not a safety system. Monitoring how people respond to an AI system is as much a control as monitoring the system itself.

6. The output belonged to someone else

In November 2022 Air Canada’s website chatbot told Jake Moffatt he could claim a bereavement fare after travelling. The airline’s policy said otherwise, and it refused the refund, arguing that it could not be held liable for information provided by its agents or representatives, including a chatbot. The British Columbia Civil Resolution Tribunal disagreed: “It should be obvious to Air Canada that it is responsible for all the information on its website.” The award was C$650.88. The principle is worth far more.

The same principle runs the other way along the supply chain. In 2025 Deloitte Australia agreed to repay part of its fee for an A$440,000 review delivered to a government department, after an academic found non-existent references and a fabricated quotation attributed to a Federal Court judgment. Generative AI had been used in preparing part of the report, and this was disclosed only in the corrected version. The errors were caught not by the client’s acceptance checks but by an outside reader after publication.

What the six have in common

Set the cases side by side and the pattern has little to do with technology. In almost every one, a formal process existed: a procurement step, an approval, a reviewer, a policy. What was missing was a control that tested the thing that mattered — outcomes for the people affected, the meaning of the target, the authority of the reviewer, the performance of a bought-in model — before and after deployment.

That is why I am wary of governance programmes that measure themselves by the documents they produce. ISO/IEC 42001 requires an AI system impact assessment, and from December 2027 the EU AI Act will require human oversight of high-risk systems that is designed with automation bias in mind. Both matter. But a control that has never been tested against an outcome is a statement of intent. The difference between governance on paper and governance that works is whether anyone asked the questions in this article, and acted on the answers.

The good news is in the same observation. If these failures had needed new mathematics, most organisations could do little about them. They did not. They needed six ordinary controls, applied with intent. Every organisation using AI can start this week: take the register of systems that affect people’s liberty, health, livelihood or legal position, and put the six questions to each one. The systems nobody can answer for are where your next case study is waiting.

This article is part of the AI Governance in Practice series. It draws on the Enterprise Intelligence Center report When Algorithms Err (Version 2.0, October 2026), which lists full sources for every case.