Risk management for Claude agents in finance

Every financial-services team that ships an AI agent eventually asks the same uncomfortable question: when this thing fails — and it will fail — how much damage can it do before someone stops it? That is the whole discipline of risk management for agentic AI, and it is the part most teams underinvest in because failures are invisible in the demo. A Claude agent that correctly resolves 9,800 disputes and silently mishandles 200 of them looks like a success on the dashboard and a problem in the regulatory exam.

Verifiable AI is not about preventing all failures. It is about making failures bounded, detectable, and recoverable. This post walks through the failure scenarios that actually occur with Claude-based agents in finance, how to think about blast radius, and the concrete containment patterns that keep a bad turn from becoming a bad quarter.

The four failure modes that actually happen

In practice, agent failures in financial services cluster into four shapes. The first is the confident wrong answer: the agent reaches a clean, well-formatted conclusion that is simply incorrect — it approves a transaction that should have been flagged. The second is tool misuse: the agent calls the right Model Context Protocol tool with the wrong arguments, or calls a write operation when it should have called a read. The third is scope creep: the agent, trying to be helpful, takes an action outside the task it was given — refunding a fee nobody asked it to touch. The fourth is cascade: in a multi-agent setup, an orchestrator trusts a subagent's flawed output and amplifies it across many cases.

Each has a different containment strategy, and conflating them is why generic "add a human in the loop" advice underperforms. A human reviewing confident wrong answers needs ground truth they often lack. A human cannot meaningfully review tool misuse at machine speed. The right control depends on the failure mode.

Blast radius: the single most important design number

Before you ship, you should be able to state your agent's blast radius in one sentence: the maximum harm a single unsupervised agent decision can cause. If a Claude agent can issue refunds, the blast radius is the largest refund it can issue without a human. If it can only draft a refund for human approval, the blast radius is near zero — it can waste a reviewer's time but cannot move money.

flowchart TD
  A["Agent proposes action"] --> B{"Action value > threshold?"}
  B -->|No| C["Auto-execute & log"]
  B -->|Yes| D["Hold for human approval"]
  C --> E{"Anomaly detector trips?"}
  E -->|No| F["Done, audit trail written"]
  E -->|Yes| G["Auto-pause agent & alert"]
  D --> H["Reviewer approves or rejects"]
  G --> H

The diagram captures the core idea: the threshold is your blast-radius dial. Cheap, reversible, common actions flow through automatically with logging. Expensive, irreversible, or anomalous actions divert to a human. The anomaly detector is the backstop for the case where a low-value action is part of a high-volume attack — a thousand small refunds that individually fall under the threshold but together are a fraud pattern. Designing this gate is not an afterthought; it is the first architectural decision.

Containing the confident wrong answer

The confident wrong answer is the most dangerous because it is the hardest to detect from output alone — by definition it looks right. The containment strategy is not to inspect the answer but to inspect the reasoning and the evidence. Require the Claude agent to emit, alongside its decision, the specific records it relied on and the policy rule it applied. Then run cheap consistency checks: does the cited policy actually apply to this customer segment? Does the cited transaction exist and match the amount? Many confident wrong answers cite evidence that does not hold up to a mechanical check, even when the prose is flawless.

The second layer is disagreement detection. Run a second, independent pass — a different prompt, sometimes a different model tier such as Haiku as a cheap cross-check against Opus — and flag cases where they disagree for human review. Disagreement does not tell you who is right, but it is a strong signal of difficulty, and concentrating human attention on the cases where two passes disagree is far more efficient than random sampling.

Bounding tool access so misuse cannot escalate

Tool misuse is contained at the tool boundary, not in the prompt. The principle is least privilege applied to MCP servers: an agent should hold only the tools its task requires, and those tools should enforce their own limits regardless of what the agent asks. A balance-lookup tool returns balances and cannot transfer. A refund tool caps the amount server-side and rejects anything above it, so even a fully compromised agent cannot exceed the cap.

This matters because the prompt is not a security boundary. You cannot reliably instruct a model never to do something dangerous and treat that instruction as a control. The control is the tool that physically cannot perform the dangerous action, or that requires a second factor — a human approval, a co-signing service — before it will. In finance this is the difference between a near miss and an incident.

Containing cascades in multi-agent systems

Multi-agent systems multiply capability and risk together. When an orchestrator delegates to subagents, a flaw in one subagent's output can propagate if the orchestrator trusts it uncritically. The containment pattern is to treat subagent outputs as untrusted input: the orchestrator validates structure and plausibility before acting on a subagent's result, the same way you would validate input from an external API. A subagent that returns a malformed or out-of-range value should trigger a retry or an escalation, never silent acceptance.

Because multi-agent runs consume several times more tokens and touch more tools than a single agent, they also have a larger natural blast radius. Use them deliberately. A good rule: if a single well-scoped agent can do the job, the reduced surface area is worth more than the marginal capability of a multi-agent design, especially early in a system's life when you still trust it least.

The recovery plan you write before launch

Containment is incomplete without recovery. Before an agent goes live, write down three things: how you detect that it has been misbehaving (the alert), how you stop it immediately (the kill-switch that disables the agent or revokes its tool access in seconds), and how you remediate the cases it already touched (the replay and correction process). Teams that skip this discover during their first incident that they can see the problem but cannot quickly stop it, which turns a contained failure into a public one.

The audit trail is what makes recovery tractable. If every agent decision is reconstructable — inputs, tool calls, policy applied, approval status — then when something goes wrong you can query "every decision in the last six hours that used this flawed assumption" and remediate exactly that set, rather than guessing or re-reviewing everything. Verifiability and recoverability are the same property viewed from two angles.

Frequently asked questions

How do I set the right blast-radius threshold?

Start far more conservative than feels necessary — route almost everything to human review — and loosen the threshold as your evals and shadow runs build evidence that the agent handles a given case class safely. It is cheap to relax a tight threshold and expensive to recover from a loose one.

Is a kill-switch really necessary if our evals are good?

Yes. Evals reduce the probability of failure; they do not eliminate it, and they cannot anticipate novel inputs, adversarial users, or upstream data corruption. The kill-switch is what bounds the worst case regardless of how the failure arises. Treat it as non-negotiable infrastructure.

Does adding all these controls slow the agent down too much?

Most controls are cheap. Structured outputs, tool-side limits, and logging add negligible latency. The expensive control is human review, which is exactly why the threshold design matters — you reserve human attention for the high-blast-radius and disagreement cases, and let the bulk flow through automatically.

Bringing agentic AI to your phone lines

Bounded blast radius and instant kill-switches matter just as much on a live call as in a back-office workflow. CallSphere builds voice and chat agents that act within strict tool limits and leave a full trail of every action. See how at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.