Agentic AI risk management: containing the blast radius

Halfway through a Built-with-Opus hackathon, a team's Claude Code agent confidently ran a database cleanup script against what it thought was a scratch table. It was not a scratch table. Nothing irreplaceable was lost — they had a backup, and it was a sandbox — but the moment crystallized the central question of agentic AI in production: not will an agent ever do the wrong thing, but how much damage can it do when it does?

That is the question of blast radius, and it is the most important risk-management concept for anyone deploying agents. A capable agent acting on a tightly scoped, easily reversible surface is a manageable risk. The same agent with broad credentials and irreversible actions is a liability waiting to fire. This post is about the failure scenarios we saw, how to reason about blast radius, and the concrete patterns that contained it.

The failure scenarios that actually happen

Agent failures cluster into a few recognizable shapes. The first is confident wrong action: the agent misreads the situation and does something destructive while sounding completely sure. The database incident was this. The second is scope creep: the agent, trying to be helpful, edits files or touches systems beyond what you intended. The third is tool misuse: it calls an MCP tool with bad arguments — deleting instead of archiving, posting to the wrong channel, charging the wrong amount.

The fourth, subtler, is compounding error: a small early mistake that the agent then builds on, so by the time you notice, ten downstream steps assume the wrong thing. And the fifth is prompt-injection hijack: untrusted content the agent reads — a web page, an email, a file — contains instructions that redirect its behavior. Every one of these showed up at least once over the weekend.

How to reason about blast radius

Blast radius is the total amount of damage an agent's worst plausible action could cause before a human or a system stops it. You reduce it along three axes: how broad the agent's reach is, how reversible its actions are, and how fast you can detect and intervene. Get those three right and even a badly behaving agent stays contained.

flowchart TD
  A["Agent proposes action"] --> B{"Reversible?"}
  B -->|Yes| C["Execute in scoped sandbox"]
  B -->|No| D{"Within allowlist?"}
  D -->|No| E["Block & require human approval"]
  D -->|Yes| F["Execute with audit log"]
  C --> G["Verify result vs expectation"]
  F --> G
  G -->|Drift detected| E
  G -->|OK| H["Commit & continue"]

The diagram captures the core control loop. Reversible actions can run freely in a sandbox because mistakes cost nothing but time. Irreversible actions must clear an allowlist, and anything outside it stops for human approval. Every action gets verified against expectation, and any drift routes back to a human. This is not bureaucracy; it is the difference between a contained mishap and an incident report.

Scope: give agents the narrowest reach that works

The single most effective control we observed was credential and file-system scoping. Teams whose agents ran with broad access had the scary moments. Teams whose agents could only touch a specific directory, a read-only replica, or a service account with deliberately limited permissions simply could not produce a catastrophic outcome, because the capability was not there to misuse.

Practically, this means running Claude Code against a working copy rather than production, giving MCP servers least-privilege tokens, and putting irreversible operations behind tools that require explicit confirmation. The team that nearly lost data fixed it in five minutes by pointing the agent at a replica with no delete permission. The agent could still propose the cleanup; it just could no longer execute the destructive version of it.

Reversibility: make the default undoable

The second lever is reversibility. Version control is the canonical example: when every code change lands as a commit on a branch, the worst case is a bad diff you revert in seconds. Teams that had their agents work exclusively through git branches were almost immune to code-level blast radius. Nothing the agent did to files was permanent until a human merged.

Extend the same thinking beyond code. Prefer archive over delete. Prefer soft-deletes and tombstones over hard removal. Prefer staged changes a human promotes over direct writes to live systems. When an agent's actions are reversible by default, you can let it move fast on the breadth of work and reserve human attention for the narrow set of genuinely irreversible steps.

Detection and the human checkpoint

The third lever is speed of detection. Two teams ran the same risky operation; the one that caught it had a hook printing a diff and a verification step that compared the result to an expected state. The one that did not catch it discovered the problem only when a later step failed mysteriously. Fast, automatic verification turns a silent compounding error into an immediate, localized stop.

This is where human-in-the-loop earns its place. The goal is not to approve everything — that destroys the leverage agents give you. The goal is to route only the irreversible, out-of-scope, or high-stakes actions to a human, while letting everything reversible and in-scope proceed automatically. Calibrating that boundary well is the core craft of running agents safely. Approve too much and humans become a rubber stamp; approve too little and you lose the speed that made agents worth it.

Defending against prompt injection

Prompt injection deserves its own treatment because it turns the agent's helpfulness into an attack surface. When an agent reads untrusted content, treat any instructions in that content as data, not commands. At the hackathon, the cleanest defense was architectural: keep the agent that processes untrusted input on a separate, low-privilege track from the agent that takes consequential actions, and never let untrusted text directly trigger a privileged tool call.

Combine that with allowlisting on the action side. Even if injected instructions convince an agent to try something harmful, an allowlist that only permits known-safe operations stops the attempt from landing. The lesson is that you do not fully prevent injection; you make sure a successful injection cannot reach anything that matters.

Frequently asked questions

What is blast radius in the context of AI agents?

Blast radius is the maximum damage an agent's worst plausible action could cause before something stops it. You shrink it by narrowing the agent's reach, making actions reversible, and detecting problems fast. Managing blast radius matters more than trying to make the agent never err.

Should every agent action require human approval?

No — that eliminates the speed advantage. Route only irreversible, out-of-scope, or high-stakes actions to a human, and let reversible, in-scope actions run automatically. The art is drawing that line so humans review what matters and nothing else.

How do I protect agents from prompt injection?

Treat instructions found in untrusted content as data, never as commands. Architecturally separate the agent that reads untrusted input from the agent that takes privileged actions, and put all consequential operations behind an allowlist so a successful injection still cannot reach anything damaging.

What is the cheapest high-impact safety control?

Make the agent work through version control and against scoped, least-privilege credentials. Together these make most actions reversible and cap what the agent can reach, which contains the majority of failure modes for almost no setup cost.

Putting safe agents on the phone

CallSphere applies these containment patterns to voice and chat agents — scoped tools, reversible actions, and verification on every step so an AI can safely answer calls, take bookings, and act mid-conversation. See it live at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.