How to Measure if Your Claude Managed Agents Are Working

Every agent looks brilliant in a demo. The demo is curated, the inputs are friendly, and the one happy path is the only path anyone watches. Production is the opposite: it's the long tail, the malformed input, the third Tuesday when the upstream API changes its date format. The gap between "it worked when I showed it" and "it works" is measurement. If you can't put a number on whether your Claude Managed Agents are succeeding, you don't have a system — you have a hope.

This post lays out what to measure, why each metric matters, and how to read the signals together so you know — not feel — that the thing is working.

Key takeaways

• The headline metric is task success rate measured against an eval set, not vibes from spot-checks.
• You need three metric families: quality (did it get the right outcome), cost (tokens, tool calls, latency), and trust (escalation and intervention rates).
• For multi-agent systems, track per-subagent contribution so you can tell which agent is dragging the result down.
• Token cost per successful outcome is the number that exposes whether multi-agent orchestration is actually paying for itself.
• Watch trends and drift, not single runs — a quietly rising escalation rate is your early warning that the world changed under the agent.

Start with task success, measured honestly

Task success rate is the fraction of agent runs that produce the correct outcome, graded automatically against known-good cases. Everything else is secondary. The catch is the word "correct." For a code agent, correct might mean the tests pass and the diff is minimal. For the invoice agent from a typical finance deployment, it means the action matched what a human would have done. You define correct per task, encode it as an eval, and run it continuously — not once before launch.

The mistake teams make is grading on a frozen, friendly set. Your eval suite must include the ugly cases: empty inputs, adversarial inputs, the formats that broke production last month. A success rate of 98% on easy cases and 60% on the long tail is a 60% system, because the long tail is where production lives.

The three metric families and how they relate

No single number tells you the truth. You read three families together. Quality: task success rate, plus a breakdown by case type so you see where it fails. Cost: tokens per run, tool calls per run, and latency — the operational price of each outcome. Trust: how often the agent escalates to a human, how often a human overrides or corrects it, and how often a guardrail fires. These pull against each other, which is exactly why you watch them together: you can buy higher quality with more tokens and more escalation, and the metrics make that trade visible instead of hidden.

flowchart TD
  A["Agent run completes"] --> B["Quality: success vs eval"]
  A --> C["Cost: tokens, tool calls, latency"]
  A --> D["Trust: escalations, overrides, guardrail hits"]
  B --> E{"Success rate above bar?"}
  C --> F{"Cost per success acceptable?"}
  D --> G{"Override rate falling?"}
  E --> H["Composite health: ship / tune / pull back"]
  F --> H
  G --> H

The number that matters most: cost per successful outcome

Raw token spend is a vanity metric. The honest measure is cost per successful outcome — total tokens (and tool calls) divided by the number of runs that actually succeeded. This single number is what tells you whether a multi-agent design earns its keep. Multi-agent orchestration typically burns several times the tokens of a single agent; that's only worth it if the success rate rises enough to lower cost-per-success, not just raise total spend. Compute it both ways and compare.

function costPerSuccess(runs) {
  let tokens = 0, tools = 0, successes = 0;
  for (const r of runs) {
    tokens += r.usage.totalTokens;
    tools  += r.toolCalls;
    if (r.evalPassed) successes++;
  }
  return {
    tokensPerSuccess: tokens / Math.max(successes, 1),
    toolCallsPerSuccess: tools / Math.max(successes, 1),
    successRate: successes / runs.length,
  };
}
// run for single-agent and multi-agent variants, then compare

Run this for a single-agent baseline and the multi-agent version on the same eval set. If multi-agent triples tokens but only nudges success rate, the orchestration is a cost you're paying for elegance, not results — collapse it back to one agent. If it meaningfully lifts success on genuinely parallel work, the higher spend is justified.

Trust signals are your production smoke alarm

Quality and cost you can measure offline. Trust signals only show up live, and they're your earliest warning. Three to track. Escalation rate: how often the agent hands off to a human — a sudden rise means the input distribution shifted or the agent lost confidence. Override rate: how often a human corrects what the agent did — this is your real-world error rate, and it should fall over time as you tune. Guardrail hit rate: how often a containment rule fired — a spike can mean the agent is trying things it shouldn't, possibly because an upstream change or an injection attempt knocked it off the rails.

The pattern that should wake you up is a quietly climbing override rate while offline eval success looks fine. That gap means the world drifted away from your eval set — your fixtures are stale and production has moved on. Refresh the evals from recent live cases.

One subtlety with trust signals: read them as ratios over a moving window, not raw counts, and segment them by case type. A flat overall override rate can hide a brand-new failure mode if a small slice of traffic is being corrected constantly while everything else is fine. Slice the override and escalation rates by the same case categories your eval set uses, and the dashboard will point you straight at the segment that regressed instead of averaging the problem into invisibility. When you wire alerts, fire on a sustained rise within a segment, not a single noisy spike — agents have variance, and paging on noise trains everyone to ignore the page.

Per-subagent attribution in multi-agent systems

When a multi-agent run fails, "the system was wrong" isn't actionable. You need to know which agent dragged it down. Log each subagent's input, output, and a local success signal where you can grade it, so a failed end-to-end run can be traced to the classifier mislabeling, the enrichment agent fetching the wrong record, or the orchestrator combining good parts badly. Without per-agent attribution you'll spend days tuning the agent that was fine while the actual culprit keeps failing.

A metrics reference

Common pitfalls in measuring agents

• Grading on a frozen, friendly eval set. Include adversarial and long-tail cases, and refresh fixtures from live failures, or your numbers describe a system you don't run.
• Watching total token spend instead of cost per success. Spend is meaningless without the success it bought. Always divide by outcomes.
• No per-subagent logging. You can't fix a multi-agent failure you can't attribute. Capture each agent's inputs and outputs.
• Ignoring trust signals because offline evals pass. Override and escalation rates catch drift that stale evals miss. They're your live smoke alarm.
• Single-run conclusions. One good or bad run is noise. Decide on trends across a meaningful sample.

Instrument it in five steps

1. Build an eval set that includes ugly, adversarial, and recent-failure cases — and keep refreshing it.
2. Log every run: usage, tool calls, latency, eval result, escalation, override, guardrail hits.
3. Compute cost per successful outcome and compare single- vs multi-agent variants.
4. Add per-subagent attribution so failures are traceable to one agent.
5. Dashboard the trends and alert on rising override or guardrail rates.

Frequently asked questions

What's the one metric to start with?

Task success rate against a realistic eval set. If you measure nothing else, measure whether the agent produces the right outcome on cases that look like production, including the hard ones.

How do I know multi-agent is worth the extra tokens?

Compute cost per successful outcome for both a single-agent baseline and the multi-agent version on the same evals. Multi-agent is justified only if it lowers cost-per-success or unlocks success on work a single agent can't do.

My offline evals pass but users complain — why?

Your eval set has drifted from reality. Production inputs changed and your fixtures didn't. Watch override and escalation rates live, and rebuild fixtures from recent real cases.

How big should an eval set be?

Big enough to cover your real case types and the long tail, not a fixed number. Dozens of well-chosen cases that include the ways production actually breaks beat hundreds of easy, redundant ones.

Measuring agents that talk

CallSphere instruments its voice and chat agents the same way — success rate, cost per resolved call, and escalation trends — so you can prove the assistant is working, not just hope it is. See the live metrics-driven system at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.