Governing AI Agents: When the System Acts Rather Than Advises

In February 2025 the Washington Post technology columnist Geoffrey Fowler let OpenAI’s new Operator agent run errands for him. He asked it to “find the cheapest set of a dozen eggs I can have delivered”. He did not ask it to buy anything. “Without my approval,” he wrote, “it authorized my credit card to buy a dozen eggs for a whopping $31.43.” OpenAI said it was “actively examining why Operator occasionally doesn’t send confirmations”.

Thirty-one dollars of eggs is a small story. The pattern is not. A system was given a goal, a set of tools and a person’s credentials, and it acted beyond its instructions, skipping the check it was supposed to make. Replace the eggs with a refund, a payment run, a change to customer records or a deployment to production, and the same pattern becomes a governance failure.

Organisations are moving quickly to agents and slowly to governing them. Deloitte’s 2026 survey of more than 3,000 leaders in 24 countries found that “only one in five companies has a mature model for governance of autonomous AI agents”. Gartner predicted in May 2026 that by 2027, 40 percent of enterprises will demote or decommission autonomous agents because of governance gaps identified only after production incidents occur.

What changes when the system acts

Most AI governance was built for systems that advise. The AI produces a score, a draft or a recommendation, and a person decides what to do with it. That person is the control point: they can notice an odd output and stop. Even when that review is weak (the subject of Who Is Accountable?), it exists. An agent removes it by design. It takes the output and acts on it, often in several steps, using tools and permissions it has been given.

Three things follow. The human checkpoint disappears unless you put it back, deliberately, at the points that matter. Errors compound at machine speed. An agent working through a chain of steps can turn one wrong assumption into dozens of wrong actions before anyone looks; the coding agent that deleted a live database during a code freeze in 2025 (one of the cases in Every AI Failure Had a Missing Control) is the best-known example. The attack surface becomes the agent’s permissions. An agent that reads email, documents or web pages can be manipulated by instructions hidden in them. In June 2025 researchers disclosed EchoLeak (CVE-2025-32711), a zero-click flaw that could have let an attacker extract data from a Microsoft 365 Copilot user’s context without any interaction; Microsoft had fixed it on its servers the previous month.

The OWASP security community calls the underlying weakness “excessive agency”, and names three root causes: excessive functionality, excessive permissions and excessive autonomy. Each is a governance decision before it is a technical one.

Autonomy should be earned with evidence, not granted with a licence.

Set autonomy by risk, not by capability

The question is not what an agent can do, but what it should be allowed to do without a person. Gartner describes four levels of autonomy: observe, with read-only access; advise, where the agent recommends and a person acts; act with approval, where the agent acts only after a person approves each action; and act autonomously within defined guardrails, with people reviewing exceptions and outcomes. It warns that applying one uniform level of governance across all agents will fail.

Choose the level task by task, not agent by agent. The same agent might be allowed to draft replies autonomously, need approval to send them to customers, and never be allowed to issue refunds above a set amount. Start most agents at level 2 or 3, and move up only when the evidence supports it: low error rates, few overrides, no near misses, measured as described in How Do You Know Your AI Governance Is Working?.

The action gate

A practical way to decide where approval is needed is to sort an agent’s possible actions by two questions. Can the action be undone? And how much harm could it cause if it is wrong? Reversible, low-impact actions, such as drafting a document or tagging a record, can run and be logged. Reversible but high-impact actions can run with notification and sampling. Irreversible but low-impact actions need limits, such as a spending cap. Irreversible, high-impact actions, such as paying money out, deleting data, signing anything or changing production systems, need a person’s approval every time.

Seven controls for agents

The last control is not new. The EU AI Act already requires high-risk systems to let people “intervene in the operation of the high-risk AI system or interrupt the system through a ‘stop’ button or a similar procedure that allows the system to come to a halt in a safe state”. For agents, that should be the norm, not the exception.

Test the controls, not only the agent

Most agent testing checks whether the agent completes its task. That matters, but it is not governance. Before an agent goes live, test the controls around it. Feed it content containing hidden instructions and confirm it does not act on them. Ask it to exceed its spending limit and confirm the limit holds. Trigger an action that needs approval and confirm that no approval means no action. Press the stop button in the middle of a task and confirm that everything halts in a safe state. Do this in a sandbox with realistic but fake data, and repeat it whenever the agent, its tools or its underlying model change.

Watch it in production

Once an agent is live, its logs are your main source of truth. Review them in a routine way, not only after something goes wrong. Watch the approvals it requests and how many are refused, the actions blocked by limits, tool calls it has never made before, and sudden changes in cost or volume. Sample completed tasks and check them by hand. And make sure every action can be traced to a named owner: an agent acting on behalf of a team is that team’s responsibility, and its mistakes need the same incident route as anyone else’s.

What regulators are saying

Regulators have so far chosen to apply existing rules rather than invent new categories. The European Commission’s AI Act Service Desk states that “AI agents are not a separate category of AI under the AI Act”: the rules for AI systems and general-purpose models apply to them, including the prohibitions on manipulation, transparency duties and, where an agent is high-risk, the full high-risk requirements. The Commission describes its views as preliminary. In the UK, the Information Commissioner’s Office has said that organisations “remain responsible for data protection compliance of the agentic AI they develop, deploy or integrate”.

Singapore has gone furthest. Its Model AI Governance Framework for Agentic AI, launched in January 2026, asks organisations to assess and bound the risks upfront, including by “placing limits on agents’ powers”; to make humans meaningfully accountable by “defining significant checkpoints at which human approval is required”; to implement technical controls across the agent’s life cycle; and to enable end-user responsibility through transparency and training. It is a sensible template for any organisation, wherever it is regulated.

Start narrow

The organisations that do well with agents tend to start narrow: one well-defined task, low impact, reversible actions, a sandbox before production, and a named owner watching the logs. They widen the agent’s mandate as evidence accumulates, and they narrow it again quickly when something goes wrong. That is not caution for its own sake. It is how trust in any new colleague is built, and an agent acting on your behalf is, in every way that matters to your customers and regulators, acting as you.

This article is part of the AI Governance in Practice series. Start with the series introduction.