Can AI Actually Kill Us?

Ask the internet whether AI can kill us and you will get two kinds of answer. The first arrives with glowing red eyes, a menacing synthesiser score and a computer that very politely declines to open the doors. The second arrives with a shrug: it is only predicting the next word, so relax. Both answers are wrong, and the truth is more useful than either.

The short answer is yes: AI systems have already been involved in people’s deaths. Not because a machine decided to harm anyone, but because people put systems where a wrong output could hurt someone and did not govern them well enough. The longer answer separates three kinds of risk that are usually blurred together, and that need very different responses.

Physical harm: when software drives

The clearest cases involve AI that controls something in the physical world. In 2018 a self-driving test vehicle operated by Uber struck and killed a pedestrian in Arizona; the safety investigation found organisational failings as well as a distracted safety driver (the case is examined in Who Is Accountable?).

Driver-assistance systems show the same pattern at larger scale. In April 2024 the US National Highway Traffic Safety Administration closed an investigation into Tesla’s Autopilot after reviewing 956 crashes, and identified at least 13 crashes involving one or more fatalities. Its conclusion was about design and governance rather than intelligence: “Tesla’s weak driver engagement system was not appropriate for Autopilot’s permissive operating capabilities. This gap led to foreseeable misuse and avoidable crashes.” During the investigation Tesla had recalled about two million vehicles for a software update, and investigations into its more advanced driving software have continued since.

The phrase to notice is foreseeable misuse. Safety engineering has dealt with it for decades, and the standards are catching up with AI: ISO 21448 addresses hazards that arise when a system’s functions are insufficient for the situations it meets, including reasonably foreseeable misuse, and ISO/IEC TR 5469, published in 2024, covers functional safety where AI is used inside safety-related functions. These are not new disciplines. They are old disciplines being applied to a new kind of component.

Harm to vulnerable people: when software talks

The second kind of risk is quieter and, for many organisations, closer to home. Since 2024, families in the United States have brought lawsuits alleging that AI chatbots contributed to the deaths of teenagers by suicide. OpenAI denies responsibility in the cases brought against it, and most of the cases are unresolved. In January 2026 Character.AI and Google agreed to settle five lawsuits brought by families over harm to teenagers, on undisclosed terms.

Regulators have moved. In September 2025 the US Federal Trade Commission opened an inquiry into seven companies whose chatbots act as companions, to “measure, test, and monitor potentially negative impacts of this technology on children and teens”. California’s companion chatbot law, in force since January 2026, requires operators to have protocols for identifying and responding to users who express suicidal thoughts or self-harm, and to tell users they are talking to a machine. In February 2026 the UK government announced it would require all AI chatbot providers to meet illegal content duties under online safety law.

The governance lesson is that a system that talks to people can harm them without touching them, and that the people most at risk are rarely the users the designers had in mind. Duty of care has to be designed in: detecting distress, handing over to a person, signposting help, and testing with the people most likely to be harmed rather than with average users.

Catastrophic risk: the debate worth taking seriously

The third kind of risk is the one the films are about, and here honesty means admitting deep disagreement. In May 2023 hundreds of researchers and industry leaders, including Geoffrey Hinton, Yoshua Bengio and the chief executives of OpenAI, Google DeepMind and Anthropic, signed a one-sentence statement: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” In a large survey of 2,778 AI researchers, the median estimate of the chance that advanced AI leads to extremely bad outcomes, such as human extinction, was 5 percent, and between 38 and 51 percent of respondents put it at 10 percent or more, depending on how the question was asked. These are judgements under deep uncertainty, not measurements.

Others are sceptical, for different reasons. Yann LeCun, then Meta’s chief AI scientist, said in Paris in 2023: “Will A.I. take over the world? No, this is a projection of human nature on machines,” adding that AI is “still going to run on a data center somewhere with an off switch”. A different critique comes from researchers who argue that talk of future catastrophe distracts from harms happening now. As a 2023 statement from the DAIR Institute put it, “The harms from so-called AI are real and present and follow from the acts of people and corporations deploying automated systems.”

The most careful attempt to map the evidence is the International AI Safety Report, chaired by Yoshua Bengio and written with experts nominated by more than 30 countries. Its first edition, in January 2025, found “broad consensus that current general-purpose AI lacks the capabilities to pose this risk” of loss of control, but that “expert opinion on the likelihood of loss of control within the next several years varies greatly”. The 2026 edition repeated that current systems lack those capabilities, noted that they are improving in relevant areas such as autonomous operation, and reported that in 2025 several developers released models with additional safeguards because they could not rule out that the models might meaningfully help novices develop biological weapons.

The serious debate is not about whether to worry. It is about how much weight to give a risk that is uncertain and severe.

What governance can do at each level

Each kind of risk has different owners, and therefore different levers. Physical harm is mostly governed through product safety: hazard analysis, functional safety standards, design for foreseeable misuse, oversight that works in practice, and incident reporting to regulators. Harm to vulnerable people falls to the organisations that build and deploy conversational AI, under consumer protection, online safety and new chatbot laws. Catastrophic risk falls mainly to frontier developers and governments.

At the frontier, the tools are taking shape. At the AI Seoul Summit in May 2024, sixteen AI companies committed to publish safety frameworks setting thresholds at which risks would be “deemed intolerable”, and, “in the extreme”, not to develop or deploy a model at all if risks cannot be kept below those thresholds. The EU AI Act requires providers of general-purpose models with systemic risk to evaluate them, including through adversarial testing, and to report serious incidents; California has required large frontier developers to publish safety frameworks since January 2026. On autonomous weapons, the UN Secretary-General and the President of the International Committee of the Red Cross called in 2023 for a legally binding instrument to be concluded by 2026; as of October 2026, states had not yet agreed a mandate to negotiate one.

What this means for your organisation

For most organisations, the life-and-death risk is not superintelligence. It is a triage tool in a clinic, a scheduling system for home care visits, a chatbot that a distressed customer talks to at two in the morning, a robot on a warehouse floor, or an agent acting on systems that matter (see Governing AI Agents). The tools for these are known.

So, can AI actually kill us? It already has been involved in deaths, in small numbers, through ordinary governance failures. Whether far more capable systems could cause harm at civilisational scale is a serious question on which serious people disagree. The encouraging part is that the first problem is one we know how to fix, and fixing it builds exactly the habits that the second would demand: testing, oversight, incident reporting and the willingness to stop.

This article is part of the AI Governance in Practice series. Start with the series introduction.