Metrics for Claude Managed Agents That Prove It Works

Plenty of teams ship a Claude Managed Agent, watch it do impressive things in a demo, and then have no idea whether it is actually working in production. "It feels good" is not a metric. When the agent runs your code in a self-hosted sandbox and acts through an MCP tunnel, you need numbers that tell you whether it is succeeding, how much it costs to succeed, and whether you can trust it with more autonomy. This post lays out exactly which metrics matter, how to instrument them, and how to read the signals that predict trouble before users feel it.

The framing I find most useful: an agent has three jobs — get the task right, do it efficiently, and earn trust. Each maps to a small set of metrics. Track those and ignore the vanity numbers.

Key takeaways

• Measure three things: task success (did it do the right thing), efficiency (cost, tokens, latency), and trust (corrections, escalations, autonomy rate).
• The north-star metric is task success rate against a graded eval set, not user sentiment or volume.
• Watch tokens-per-task and tool-calls-per-task as early-warning signals; a sudden rise usually means the agent is struggling or looping.
• Track the human correction rate over time — falling corrections are the clearest evidence you can safely grant more autonomy.
• Instrument at the MCP server and the sandbox, because that is where the ground-truth signals live.

Task success: the metric everything else serves

Before efficiency or cost means anything, you need to know whether the agent is doing the right thing. That requires a graded eval set — a collection of representative tasks with known correct outcomes — that you run on every meaningful change. Task success rate is the fraction of that set where the agent took the correct action. This is your north star, and without it every other number is decoration.

The discipline is to grade outcomes, not vibes. For a support agent that means "did it retry the right job and say the right thing," scored against the known-correct answer, not "did the reply read nicely." For a data agent it means "did the query return the right rows." Build the set from real historical cases so it reflects the messy distribution production will actually throw at the agent.

A definition worth quoting: task success rate is the proportion of evaluated tasks on which the agent produces the correct, policy-compliant outcome — and it is the single metric that should gate every release of a managed agent.

Efficiency: cost, tokens, latency, and tool calls

An agent can be correct and still be a bad deal if each task burns a fortune in tokens or takes two minutes. Efficiency metrics tell you whether the capability is economically sane and fast enough to be useful. The four to track are cost-per-task, tokens-per-task, tool-calls-per-task, and end-to-end latency.

flowchart TD
  A["Agent run completes"] --> B["Log: tokens, tool calls, latency, cost"]
  A --> C["Grade outcome vs eval set"]
  C --> D{"Correct?"}
  D -->|Yes| E["Success metric +1"]
  D -->|No| F["Failure: capture trace"]
  B --> G{"Tokens/task rising?"}
  G -->|Yes| H["Investigate looping/struggle"]
  E --> I["Dashboard & trend"]
  F --> I
  H --> I

Of these, tokens-per-task and tool-calls-per-task are the most diagnostic. A healthy agent settles into a stable band — say it usually resolves a task in a handful of tool calls. When that number jumps, the agent is usually struggling: retrying, second-guessing, or looping. Multi-agent setups deserve extra scrutiny here, because orchestrator-subagent runs often consume several times more tokens than a single agent, so the efficiency bar must justify the coordination cost.

Cost-per-task is the number your finance partner will ask about, but treat it as derived rather than primary. It moves when pricing changes, when you switch models, or when the agent's behavior drifts, so it conflates three different stories. Tokens-per-task isolates the behavior story, which is the one you can actually fix in a prompt or a tool. Watch cost for the budget conversation and tokens for the engineering conversation, and you will rarely be surprised by either.

Latency matters most for anything a human or customer waits on. For background batch tasks, throughput and cost dominate and latency is secondary. Decide which regime you are in and weight accordingly.

Trust signals: the metrics that unlock autonomy

The most important business question about an agent is "can we let it act without a human in the loop?" The metrics that answer it are trust signals: the human correction rate (how often a reviewer overrides or fixes the agent's proposed action), the escalation rate (how often the agent hands off because it could not proceed), and the autonomy rate (the share of tasks completed end-to-end without human touch).

Read these as a time series, not a snapshot. A correction rate that falls week over week is the cleanest evidence that you can safely remove an approval gate for a class of tasks. A correction rate that plateaus high means the agent has hit a ceiling on that task and needs better tools, better instructions, or a narrower scope — not more autonomy.

Pair the rates with the captured failure traces. Every correction and every escalation should leave behind a trace you can feed back into the eval set, so the thing that tripped the agent today becomes a graded test tomorrow. That feedback loop is what turns raw metrics into improvement.

Instrument where the truth lives

You cannot measure what you do not capture, and the ground-truth signals live at the boundaries: the MCP server sees every tool call and its arguments and result; the sandbox sees every run's resource use and exit status. Instrument both. The MCP server is the natural place to count tool calls, latency per tool, and error rates; the sandbox is where you capture tokens, wall-clock time, and whether the run aborted on a budget cap.

Here is the minimal shape of a per-run record worth logging — enough to compute every metric above:

{
  "run_id": "run_8c21",
  "task_type": "export_ticket",
  "outcome": "resolved",
  "graded_correct": true,
  "tokens": 14820,
  "tool_calls": 4,
  "latency_ms": 9100,
  "cost_usd": 0.18,
  "human_corrected": false,
  "escalated": false
}

With this record per run, your dashboard practically builds itself: success rate from graded_correct, efficiency from tokens and tool_calls and cost_usd, and trust from human_corrected and escalated. Keep the schema stable so trends stay comparable across model and prompt changes.

Common pitfalls in measuring agents

• Optimizing for volume. "The agent handled 10,000 tasks" says nothing about whether it handled them correctly. Lead with success rate, not throughput.
• Grading on tone instead of outcome. A fluent, confident, wrong answer is worse than a terse correct one. Score the action taken against ground truth.
• Ignoring tokens-per-task until the bill arrives. Cost drift is an early symptom of an agent struggling. Alert on it as a leading indicator, not a monthly surprise.
• Treating a one-time eval as done. The production distribution shifts; an eval set that never grows goes stale. Feed every correction and escalation back into it.
• No trend lines. A single week's correction rate is noise. The slope over weeks is the signal that tells you whether to grant or pull autonomy.

Stand up agent metrics in five steps

1. Build a graded eval set from real cases and compute task success rate on every change.
2. Log a per-run record with tokens, tool calls, latency, cost, and outcome at the sandbox and MCP server.
3. Chart tokens-per-task and tool-calls-per-task and alert when they drift upward.
4. Track correction, escalation, and autonomy rates as weekly trend lines.
5. Feed every failure trace back into the eval set so the metric sharpens over time.

Frequently asked questions

What is the single most important agent metric?

Task success rate against a graded eval set. It tells you whether the agent does the right thing, and it gates every release. Efficiency and trust metrics matter, but they are meaningless if the agent is frequently wrong.

How do I know when to give an agent more autonomy?

Watch the human correction rate as a time series. When it falls steadily and stays low for a class of tasks, you can safely remove the approval gate for those tasks. A high plateau means the agent needs improvement, not more freedom.

Why track tokens-per-task if cost is already logged?

Because tokens-per-task is a leading indicator. A sudden rise reveals the agent struggling or looping before it shows up as a meaningful cost change, giving you time to investigate. It is also independent of pricing changes, so the trend stays interpretable.

Do multi-agent systems need different metrics?

The same metrics apply, but watch efficiency harder. Multi-agent runs commonly use several times the tokens of a single agent, so the orchestration must buy a real success-rate gain to be worth it. Measure that tradeoff explicitly rather than assuming more agents is better.

Measured agentic AI on your phone lines

CallSphere instruments its voice and chat agents the same way — success rate, cost-per-conversation, and correction trends — so AI that answers every call and books work is proven, not just promised. See the live numbers behind it at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.