How to Measure if Your Claude Agent Actually Works

Ask a team whether their Claude agent is working and you often get a vibe rather than a number. "It feels good." "Customers seem happy." "We haven't had complaints." Those are not measurements — they are the absence of measurement wearing a confident expression. The hard truth of running agents in production is that without the right metrics, you cannot tell a genuinely good agent from a lucky one, and you certainly cannot tell whether last week's prompt change helped or quietly made things worse. This post is about the signals that turn that fog into something you can manage.

The core difficulty is that agents do not have a single correct output to check against. A good answer can be phrased many ways, and a bad action can look superficially reasonable. So measurement has to move up a level: from "is this output exactly right" to "did this agent achieve the outcome it was supposed to, safely, at an acceptable cost." Getting that framing right is most of the battle.

Task success is the metric that matters most

The headline metric for any agent is task success rate: of the requests the agent attempted, how often did it actually accomplish the user's goal? This sounds obvious, but defining "accomplished" rigorously is where teams differ. For an invoice agent it might be "gave a correct, grounded answer or routed correctly." For a coding agent it might be "produced a change that passes the test suite and review." The definition must be specific enough that two people scoring the same transcript agree.

Because there is no exact-match check, task success is usually measured two ways in combination. A held-out eval set gives you a controlled score you can run on every change — the lab measurement. And a sample of real production transcripts, graded by humans or a carefully designed LLM judge, gives you the field measurement. The eval set tells you whether a change is better in principle; the production sample tells you whether it is better in reality. You need both, because traffic always drifts away from your eval set over time.

The supporting signals that explain the headline

Task success tells you whether the agent is working; a cluster of supporting metrics tells you why, and they are what you actually act on day to day.

flowchart TD
  A["Agent run"] --> B["Task success?"]
  A --> C["Grounding / hallucination rate"]
  A --> D["Escalation rate"]
  A --> E["Cost & latency per outcome"]
  B --> F["Quality dashboard"]
  C --> F
  D --> F
  E --> F
  F --> G{"Trend vs baseline?"}
  G -->|Degrading| H["Investigate & add eval case"]
  G -->|Stable / improving| I["Ship next change"]

Grounding rate measures how often the agent's factual claims are backed by a tool result rather than invented. For any agent that touches real data, this is the safety-critical metric, and it should be tracked as relentlessly as success. A high task-success number masking a creeping hallucination rate is one of the most dangerous patterns in production agents.

Escalation rate — how often the agent hands off to a human — is wonderfully informative because it cuts both ways. Too high and the agent is not actually saving work; too low and you should worry it is overstepping into cases it should defer. Watching where escalations cluster also tells you exactly which capabilities to build next. Cost and latency per resolved outcome ground the whole program in economics: an agent that succeeds but costs more than the human it replaced is a science project, not a system. Measuring per outcome, not per token, keeps the focus honest.

Why averages lie and tails tell the truth

A single average hides the failures that matter. An agent with 97% task success sounds excellent until you look at the 3% and find that it includes the highest-value transactions, or that the failures are concentrated in one customer segment, or that they are all the same recurring mistake. Measurement discipline means slicing every metric by segment, by request type, and by value, and paying special attention to the tail.

A useful definition to standardize your reporting: agent task success rate is the fraction of attempted requests for which the agent achieved the user's intended outcome, judged against an explicit per-request success criterion. The phrase "explicit per-request success criterion" is the load-bearing part — without it, the number is just a feeling with a percent sign. The same rigor applies to every supporting metric: define the criterion, then measure against it consistently.

Detecting drift before users do

The most insidious failure of a production agent is not a crash; it is slow erosion. The model is stable, the code is unchanged, but the world shifts — new product lines, new phrasing from users, a new edge case that becomes common — and the agent's once-tuned behavior degrades against the new reality. Because nothing throws an error, drift is invisible unless you are watching for it.

The defense is a continuously refreshed sample of production traffic scored on the same rubric over time, so a downward trend shows up as a trend rather than a surprise. Pair it with a small set of canary cases that exercise your trickiest behaviors and run on a schedule; when a canary that always passed starts failing, you have an early warning. And feed every confirmed production failure back into the eval set as a permanent case, so the suite tracks reality instead of slowly going stale. An eval set that does not grow is one that is quietly becoming irrelevant.

Closing the loop from metric to action

Metrics only matter if they change behavior. The teams that get real value wire their signals into a tight loop: a dashboard surfaces a degrading trend, an engineer pulls the failing transcripts, the root cause becomes a new eval case, a fix is made, and the eval set confirms the fix before it ships. Every cycle makes the agent measurably better and the safety net measurably stronger. Without that loop, dashboards become wallpaper — pretty, watched briefly, and ignored. With it, measurement becomes the engine that compounds the agent's quality over time.

Frequently asked questions

What is the single most important agent metric?

Task success rate against an explicit, per-request success criterion. It directly answers whether the agent is doing its job. But it must be paired with grounding rate so a high success number cannot hide a rising tendency to fabricate facts, which is the most dangerous way an agent can look good while being unsafe.

How do I measure success when there is no single correct answer?

Move from exact-match checking to outcome judging. Define what "goal achieved" means per request, then score with a held-out eval set for controlled comparisons and a graded sample of real transcripts for field truth. Human or well-designed LLM-judge grading handles the variance that exact-match assertions cannot.

Why measure cost per outcome instead of cost per token?

Because the business cares about resolved work, not tokens. An agent that uses many tokens but reliably resolves a request can still be cheaper than a human; one that uses few tokens but rarely resolves anything is expensive in disguise. Cost per outcome ties the metric to the value actually delivered.

How do I catch quality drift early?

Continuously sample and re-score production traffic on a stable rubric so degradation appears as a trend, run scheduled canary cases over your trickiest behaviors, and add every confirmed failure to your eval set. Together these turn slow erosion into an alert you see before your users do.

Bringing agentic AI to your phone lines

CallSphere measures its voice and chat agents on exactly these signals — resolution rate, grounding, escalation, and cost per handled conversation — so every call is accountable. See the live system at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.