How to Measure if Your Claude Agent Is Working

An agent that demos well tells you almost nothing. Demos are curated, and a system that's right eight times out of ten can look flawless if you only show the eight. The real question — is this agent actually working in production? — can only be answered with measurement, and measuring agents is its own skill because the thing you're measuring makes decisions instead of just transforming inputs. You can't just diff output against a fixed expected value when there are several correct ways to accomplish a task.

Teams that don't measure end up in a familiar trap: the agent feels fine, ships, and then quietly degrades as prompts drift, data changes, or edge cases accumulate. Nobody notices until a customer complaint or a cost spike forces an investigation. The fix is to decide up front what "working" means and instrument for it, so the agent's health is a number on a dashboard rather than a vibe.

Start with task success, defined narrowly

The foundational metric is task success rate: of the tasks the agent attempted, what fraction reached a correct outcome? The trick is defining "correct" precisely for your domain. For a support-drafting agent it might mean factually accurate and sent with minimal human edits. For a data-extraction agent it might mean every required field pulled correctly. Vague success criteria produce vague metrics, so this definition is worth real debate before you measure anything.

A useful definition to anchor on: an agent eval is a repeatable test that runs the agent against known inputs and scores its outputs and actions against a rubric. Build a set of representative cases with known good outcomes and run it on every change. This offline score is your early-warning system — it catches a regression before production does, and it lets you compare two prompts or two models honestly instead of by gut feel.

The signals that matter in production

Offline evals tell you about a fixed set of cases. Production tells you about reality, and reality has a different set of signals worth watching.

flowchart TD
  A["Agent run"] --> B["Log trajectory"]
  B --> C{"Outcome?"}
  C -->|Success| D["Task success rate"]
  C -->|Human stepped in| E["Intervention rate"]
  C -->|Escalated| F["Escalation rate"]
  B --> G["Tokens & steps"]
  G --> H["Cost per task"]
  D --> I["Quality dashboard"]
  E --> I
  F --> I
  H --> I

The most honest production metric is human intervention rate: how often a person had to correct, override, or redo the agent's work. A low and falling intervention rate means the agent is genuinely carrying load; a rising one means trouble, even if the agent looks busy. Closely related is escalation rate — how often the agent correctly hands off what it can't handle. You want this neither too high (the agent is useless) nor suspiciously low (it's guessing instead of flagging).

Then there's cost and efficiency. Track tokens and tool calls per task, and watch the trend. A creeping cost-per-task often signals that the agent is taking more steps than it should — looping, re-reading context, or fumbling tool use. Cost is a quality signal in disguise: a sudden jump usually means the agent's behavior changed, and it's worth investigating why before the bill does it for you.

Trajectory quality, not just final answers

Two agents can produce the same correct answer by very different paths, and the path matters. An agent that reaches the right reply after eight redundant tool calls is fragile and expensive even when it succeeds. So beyond the final outcome, score the trajectory: did the agent pick the right tools, in a sensible order, without unnecessary detours? Reading trajectories is also how you understand why a failure happened — the final answer tells you something broke, but the step-by-step trace tells you where.

This is why full trajectory logging is non-negotiable. Every prompt, tool call, argument, and result should be captured. When a metric moves the wrong way, the logs are how you go from "intervention rate is up" to "the agent started misusing the lookup tool after we changed its description." Without trajectories you're left guessing; with them, debugging an agent feels almost like debugging ordinary software.

Watching for drift over time

The metric most teams forget is drift. An agent that scored well at launch can degrade silently as the world changes — new product features the docs don't cover, new ticket types, a model version update, or prompt edits that helped one case and quietly hurt others. The defense is to keep running your eval set on a schedule, not just at launch, and to alert when scores fall. Treat your evals like a regression suite that runs forever, because an agent's quality is never finished — it's maintained.

It also helps to keep feeding production failures back into the eval set. Every real mistake that reaches a human is a test case you didn't have before. Over time this makes your evals sharper and more representative of what the agent actually faces, so the offline score tracks production reality more and more closely.

Tying metrics to a rollout decision

Metrics earn their keep when they gate decisions. Don't widen an agent's autonomy because it feels ready — widen it when task success is consistently high, intervention rate is low and stable, escalation is conservative, and cost-per-task is flat. Conversely, when any of those move the wrong way, that's the signal to pull back, tighten a guardrail, or roll back a change. Metrics turn "I think the agent is fine" into "the agent meets the bar to handle this subset," which is the only basis on which you should ever grant an autonomous system more trust.

Frequently asked questions

What is the most important metric for an AI agent?

Human intervention rate — how often a person has to correct or redo the agent's work. It's the most honest measure of whether the agent is genuinely carrying load, because it reflects real outcomes rather than curated demos.

How do I measure success when there are many correct answers?

Score against a rubric instead of an exact-match expected value. Define what a correct outcome looks like — accuracy, right tools used, correct escalation — and evaluate both the final result and the trajectory against it.

Why track cost per task as a quality signal?

Because rising token and tool-call counts usually mean the agent is taking more steps than it should — looping or fumbling tools. A cost spike is often the first visible symptom of a quality regression, so it's worth investigating as one.

How do I catch an agent degrading over time?

Run your eval set on a schedule, not just at launch, and alert when scores drop. Feed production failures back into the eval set so it stays representative, and treat the suite like a regression test that runs forever.

Bringing agentic AI to your phone lines

The same metrics decide whether a voice or chat agent is ready for customers. CallSphere instruments its voice and chat agents on task success, escalation, and cost so quality is a number, not a hope. See it live at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.