Measuring Agentic AI Success: Metrics That Prove It Works

An agent that demos well and an agent that works are different things, and the gap between them is measurement. Plenty of teams ship a Claude agent, watch a few impressive runs, declare victory, and then have no idea three months later whether it is helping or quietly causing problems. The teams that succeed treat their agents like any other production system: they instrument them, they define what good looks like before launch, and they watch a small set of signals that actually correlate with value. This post is about choosing those signals, avoiding the vanity ones, and building a measurement loop that tells you the truth.

Key takeaways

• The headline metric is task completion rate — did the agent fully resolve the task without a human stepping in?
• Pair outcome metrics with eval pass rate as a leading indicator you can check before shipping.
• Watch escalation rate and reasons; rising escalations are an early warning, not just a cost.
• Track cost per resolved task, including tokens — multi-agent runs can quietly multiply this.
• Ignore vanity metrics like raw run count or token volume; they reward activity, not outcomes.

Start with the outcome, not the activity

The first question is deceptively simple: what does success mean for this agent? For a support agent it is a resolved ticket the customer did not have to follow up on. For a coding agent it is a merged change that passed review and did not get reverted. For a research agent it is a report the requester actually used. Notice that all of these are outcomes, and all of them are defined by a human accepting the result. Activity metrics — how many runs, how many tokens, how many tool calls — measure effort, not value, and optimizing them leads you somewhere bad.

The cleanest top-line metric most teams can adopt is task completion rate: of the tasks the agent attempted, what fraction did it fully resolve without a human having to intervene or redo the work. It is honest because it counts the silent failures — the times the agent produced something plausible that a human had to quietly fix. A demo never shows you those; a completion-rate metric does.

The metric stack

No single number is enough. The useful approach is a small stack of metrics at different layers, from leading indicators you can check pre-launch to lagging outcomes that take days to settle. The flow below shows how a signal moves from an individual run up to the metrics a team reviews.

flowchart TD
  A["Agent run completes"] --> B["Log trace: inputs, tools, output, cost"]
  B --> C{"Resolved without human?"}
  C -->|Yes| D["Count as completed"]
  C -->|No| E["Tag escalation reason"]
  D --> F["Roll up: completion rate, cost/task"]
  E --> F
  F --> G["Compare against eval pass rate"]
  G --> H["Weekly review & trend"]

The leading indicator is eval pass rate — the fraction of your test suite the current agent version passes. You can read it before shipping a change, which makes it your gate. The lagging indicators are the production outcomes: completion rate, escalation rate, and cost per resolved task. When eval pass rate is high but production completion is low, your evals are missing real-world cases — which is itself a valuable signal that your test set needs to grow.

Escalation as a signal, not just a cost

Escalation rate — how often the agent hands off to a human — is easy to read as pure overhead, but it is one of the richest signals you have, provided you tag why. An agent that escalates because it correctly recognized a case outside its scope is behaving well; that is the system working. An agent whose escalations are climbing because it keeps hitting a tool error or misreading a new ticket type is telling you something broke. The number alone is ambiguous; the reason codes turn it into a diagnostic.

Make escalation reasons a required, structured field. Over a few weeks the distribution of reasons becomes a map of where to invest: the most common reason is your next skill improvement or eval case. Teams that log escalations as a single undifferentiated count throw away most of the value.

Cost per resolved task

Raw token volume is a vanity metric, but cost per resolved task is essential, because it tells you whether the agent is economical at the only unit that matters — a completed outcome. This is where multi-agent designs deserve scrutiny: a multi-agent run typically uses several times more tokens than a single agent, so an orchestrator that spawns many subagents can quietly triple your cost per task. If the completion rate did not improve proportionally, the extra agents are not earning their keep.

Compute it simply: total token and tool cost over the period, divided by the number of tasks resolved without human intervention. Tracking the trend matters more than the absolute number. A cost-per-task that creeps up while completion stays flat is a sign of growing inefficiency — often a skill that got verbose or an agent that started over-using an expensive tool.

Watch the distribution, not just the average

Averages hide the failures that hurt most. An agent with a ninety percent completion rate sounds healthy, but if the ten percent it fails are concentrated in your highest-value tasks, the average is lying to you. The fix is to slice every metric by a segment that matters — task type, customer tier, channel — and look at the worst slice, not the headline number. A completion rate that is excellent overall but poor on a specific ticket category points you straight at the next skill improvement.

The same logic applies to latency and cost. A handful of pathological runs that loop many times before resolving can be invisible in the mean and obvious in the ninety-fifth percentile. Those tail cases are usually where the agent is struggling with something real — an ambiguous input, a flaky tool, a scenario your evals never covered. Reading the tail of the distribution is one of the fastest ways to find the next thing worth fixing, and it is exactly the view an average-only dashboard denies you.

What to measure and what to ignore

Signal	What it tells you	Use it?
Task completion rate	Real value delivered	Yes — top line
Eval pass rate	Leading, pre-ship gate	Yes — release control
Escalation rate + reasons	Where the agent struggles	Yes — diagnostic
Cost per resolved task	Economic efficiency	Yes — watch the trend
Raw run count	Activity, not value	No — vanity
Total token volume	Effort, not outcome	No — normalize per task

Build the measurement loop in five steps

1. Define what "resolved without a human" means for this specific agent, in writing.
2. Log every run as a trace: inputs, tools used, output, and cost.
3. Tag every escalation with a structured reason code.
4. Roll up completion rate, escalation reasons, and cost per resolved task weekly.
5. Compare production completion against eval pass rate and grow the eval set where they diverge.

Common pitfalls

• Optimizing activity metrics. Run count and token volume reward a busy agent, not a useful one. Always normalize to resolved tasks.
• Counting plausible output as success. If you do not track silent fixes, your completion rate is a flattering lie. Measure human intervention, not output volume.
• Logging escalations without reasons. A bare escalation count is ambiguous; the reason codes are where the diagnostic value lives.
• Trusting eval pass rate alone. A high pass rate with low production completion means your evals are out of date, not that the agent is great.
• Ignoring cost drift. A creeping cost per task with flat completion signals inefficiency — often a bloated skill or an overused expensive tool.

Frequently asked questions

What is the single best metric for an AI agent?

Task completion rate — the fraction of attempted tasks the agent fully resolved without a human intervening or redoing the work. It captures real value and exposes the silent failures that demos hide, which is why it belongs at the top of the stack.

How is eval pass rate different from completion rate?

Eval pass rate is a leading indicator you measure against a fixed test suite before shipping; completion rate is a lagging production outcome measured on live tasks. You use the first as a release gate and the second as ground truth, and divergence between them tells you your evals need refreshing.

Why track cost per resolved task instead of total tokens?

Because total tokens reward effort, not outcomes. Cost per resolved task ties spend to the only unit that matters — a completed result — and it surfaces when multi-agent designs, which use several times more tokens, are not earning their extra cost.

How often should we review these metrics?

A weekly trend review works for most teams, with eval pass rate checked on every change before it ships. The point is to watch trends, not absolute numbers — a metric drifting the wrong way over three weeks is a clearer signal than any single data point.

Bringing agentic AI to your phone lines

CallSphere instruments voice and chat agents with these exact signals — completion rate, escalation reasons, and cost per resolved call — so you can prove the agent is working, not just guess. See it live at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.