The Risk Argument Against Enterprise AI Agents Is Backwards

A strange thing is happening in enterprise AI.

The same companies that will eventually trust autonomous systems to move human beings through traffic are nervous about letting an AI agent move a record from one system to another.

At first that sounds reasonable. Enterprise systems are sensitive. They hold customer data, pricing, contracts, invoices, supplier relationships, regulated workflows, and the accumulated operational memory of the business. Nobody wants a model wandering through that environment with vague instructions and broad permissions. So the instinct is to slow down: don't let the agent reconcile the invoice, don't let it update the CRM record, don't let it touch the sourcing scenario until we are absolutely certain nothing can go wrong.

Now compare that to a self-driving car.

A self-driving car operates in the physical world, in real time, surrounded by actors nobody controls. Other drivers run red lights. Pedestrians step into crosswalks while looking at their phones. Cyclists weave between lanes. Construction crews change the road geometry overnight. Weather collapses visibility. Emergency vehicles break the normal rules and expect everyone else to adapt. Every single mile is an adversarial test of perception, prediction, planning, and control, and the cost of getting it wrong is measured in human injury, in milliseconds.

I spent time around this problem at GM, and the thing people outside the field tend to miss is this: autonomous driving is not hard because the vehicle needs to "think." It is hard because the vehicle has to act safely inside an open system where most of the risk originates outside its control.

That is the opposite of almost every enterprise AI environment.

We already deploy autonomy into harder environments

If you want a sense of how far commercial autonomy has already moved, look at the numbers.

GM's Super Cruise has now logged more than a billion hands-free miles across roughly 750,000 vehicles and two dozen models. That is a driver-assistance system (a human is still responsible and still in the loop), but the scale tells you something about how quickly trust and adoption compound once the technology proves itself.

The more striking case is fully driverless operation. Waymo's published safety data covers more than 170 million miles driven with no human behind the wheel at all, across Phoenix, San Francisco, Los Angeles, and Austin. In the areas where it operates, Waymo reports 92% fewer serious-injury-or-worse crashes compared with human driver benchmarks, along with similar reductions in crashes involving pedestrians and cyclists. This is not a lab result. It is autonomy acting, unsupervised, in one of the most adversarial environments we routinely operate in.

None of this means the technology is finished or that it operates without scrutiny: regulators are actively investigating specific incidents, and the companies involved are being held to account for them. But that scrutiny is part of the point. We did not decide autonomy was acceptable because someone promised it would be perfect. We decided it was acceptable because it could be deployed in a bounded, measured, and monitored way, and because the data could be put on the table and argued over in public.

Now hold that next to an enterprise agent.

An MCP-enabled agent operating inside a company does not need to infer whether a pedestrian is about to step off the curb. It does not need to predict whether the truck in the next lane is about to drift. It does not contend with ice, glare, road debris, or a human having the worst driving day of their life. It needs to call tools, retrieve context, respect permissions, produce work, ask for approval when required, and leave an audit trail behind it.

That is not a trivial risk. But it is a dramatically more governable one.

The concept that makes autonomy acceptable: a bounded domain

The lesson from autonomous vehicles is not that companies should deploy AI recklessly. It is that autonomy becomes acceptable the moment it is bounded.

Self-driving systems are not trusted everywhere, in every condition, for every maneuver, on day one. They operate inside what the industry calls an operational design domain: mapped geographies, supported road types, defined weather assumptions, monitored performance envelopes, fallback procedures, and a documented safety case. The autonomy is real. It is just not infinite. The entire discipline of deploying AVs is the discipline of drawing that boundary carefully and defending it with data.

Enterprise agents need exactly the same architecture.

An agent should have an operational design domain too. It should know which systems it can reach, which tools it can call, which records it can modify, which actions require human approval, which actions are simply forbidden, how its outputs are logged, how its errors escalate, and how a human steps in to take the wheel. "Autonomy" in the enterprise should never mean "unbounded." It should mean a clearly scoped set of capabilities with monitoring and a kill switch: the software equivalent of a disengagement.

This reframes the entire risk conversation. The question is not should AI agents be allowed to act? Action is not the problem. Unbounded action is the problem. The real question, the practitioner's question, is: what is this agent's operational design domain?

The blast radius is smaller than the fear suggests

The comparison only works if we are honest about consequences.

A bad decision from an autonomous vehicle can cause physical harm to a human being in a fraction of a second, in the open, with no undo. A bad decision from an enterprise agent can cause a data leak, a wrong transaction, a broken workflow, or reputational damage. Those are serious. I am not waving them away. But they are, in most cases, more observable, more reversible, and more constrainable than a collision. You can log them. You can rate-limit them. You can require approval before the consequential ones execute. You can roll many of them back.

We have collectively decided to accept the irreversible, physical-world risk profile, under the right controls. It is strange to then treat the smaller, more recoverable, software-bounded risk as categorically unacceptable.

Where MCP actually fits

This is where the Model Context Protocol gets interesting, and where I think a lot of the framing goes wrong.

MCP should not be understood as "letting the model into the enterprise." That is the wrong mental model. A better one comes straight from the vehicle: drive-by-wire. The driver's input no longer yanks a mechanical linkage directly; it passes through a structured interface between intention and action, with limits and checks in the middle. MCP is the same idea for software. It is the control plane for bounded agency: the layer where enterprise systems expose specific capabilities to a model in structured, inspectable, permissioned ways. It is precisely where the operational design domain should be defined and enforced: scoped authorization, sandboxing, provenance, policy enforcement, logging, and auditability.

A bad MCP implementation can absolutely create risk. Overbroad permissions, weak authentication, tool poisoning, prompt injection, data exfiltration, privilege escalation: these are real failure modes, and the security community is right to name them. But notice what those are arguments for. They are not arguments against agents. They are arguments against unmanaged agents. They are the reason to build the control plane well, not the reason to refuse to build it at all.

The serious security conversation around MCP has never been "never deploy." It has been: deployment requires scoping, sandboxing, provenance, policy, monitoring, and audit. That is the same vocabulary the AV industry had to invent. It is the vocabulary of bounded autonomy.

So the enterprise answer should not be "no agents." It should be "no unbounded agents."

The agent needs a safety case

This is the part enterprise AI teams should borrow most directly from autonomous systems.

Every serious agent should have a safety case. Not a theatrical compliance document. Not a slide that says "human in the loop" and calls the problem solved. A real argument, in plain language, for why this system is safe enough to operate inside its domain.

What can it access? What can it change? What can it never do? Which actions are reversible? Which ones require approval? What happens when confidence is low? What gets logged? Who reviews failures? How do we know whether performance is improving or degrading over time?

That is the difference between an agent and a demo. A demo proves the model can do the task once. A safety case proves the system can do the task repeatedly, under constraints, with known failure modes and a way to recover when something goes wrong.

The risk you're actually carrying

Here is the part that should keep leaders up at night, and it is not the governed agent.

The most dangerous AI system in your company is not the one with scoped tools, an identity, approval gates, logging, and rollback. It is the one nobody knows exists.

Employees are already pasting data into consumer tools. Teams are already wiring up unofficial automations. Analysts are already stitching together scripts, browser extensions, chatbots, spreadsheets, and APIs to get their work done faster than the official roadmap allows. Agency has already entered the enterprise. That decision was made for you. The only open question is whether it enters through architecture or through shadow IT.

A governed agent has an operational design domain. Shadow AI has none. It is the unmonitored, unscoped, unlogged version of the exact risk everyone claims to be protecting against, and refusing to provide a governed alternative does not prevent it. It guarantees it.

Self-driving cars forced the world to develop a more serious language for autonomy: operational design domains, safety cases, disengagements, telemetry, simulation, fallback behavior, continuous monitoring. Enterprise AI deserves the same seriousness. Not panic. Not blind acceleration. Seriousness.

If we can build systems that navigate public roads full of bad actors and uncontrolled risk, we can build systems that navigate enterprise workflows full of invoices, contracts, suppliers, tickets, and records. The bar should be high. It should just not be mythical.

The question was never whether the AI should be allowed to act.

The question is what its operational design domain should be.