How to Measure if Your Claude Agent Is Actually Working

Plenty of teams ship a Claude agent, watch a few impressive demos, declare victory, and then have no idea whether the thing is actually working three months later. "It feels good" is not a metric, and neither is the number of times leadership saw it do something cool in a meeting. If you have extended Claude with MCP servers and Agent Skills to do real work, you owe yourself a real answer to a hard question: is this agent delivering value, reliably, safely, and at a cost that makes sense? This post is about how to measure that — the metrics that matter, the leading signals that warn you early, and the traps that make agents look better or worse than they are.

Start with task success, defined per task

The foundational metric is task success rate: of the tasks the agent attempted, what fraction reached a correct, complete outcome. The catch is that "correct" is meaningless until you define it for your specific task. For a refund agent, success is the right refund decision with the right action taken. For a research agent, it is an answer that is accurate and properly sourced. You cannot measure success against a vague goal, so the first real work of measurement is writing down, task by task, what a good outcome looks like — ideally as a set of checkable criteria a reviewer or an automated eval can apply.

Once you have that definition, build an eval set of representative real cases with known correct outcomes and run the agent against it on every meaningful change to a tool or skill. This gives you a stable, repeatable success number instead of vibes. The single biggest measurement mistake teams make is having no fixed eval set, so every "it's working better now" is unfalsifiable. A success rate measured against a frozen set of real cases is the bedrock everything else builds on.

The metrics beyond raw success

Success rate alone hides a lot. Three other dimensions matter enough to track from day one, and they pull in different directions, which is exactly why you watch all of them together.

flowchart TD
  A["Agent run completes"] --> B["Task success rate"]
  A --> C["Autonomy rate"]
  A --> D["Cost per resolved task"]
  A --> E["Safety incidents"]
  B --> F{"All within target?"}
  C --> F
  D --> F
  E --> F
  F -->|Yes| G["Healthy: scale up"]
  F -->|No| H["Investigate via transcripts"]

Autonomy rate is the fraction of tasks the agent completes without human intervention. A high success rate with a low autonomy rate means humans are doing most of the work and the agent is an expensive assistant. As you tune tools and skills, you want autonomy to climb while success holds — that is the curve that proves real leverage. Cost per resolved task matters because agentic and especially multi-agent runs can burn several times more tokens than a single call; an agent that succeeds but costs more than the human it replaced is a science project, not a system. And safety incidents — wrong high-impact actions, near-misses caught by approval gates, injection attempts — must be counted explicitly, because a single bad refund or sent email can outweigh a thousand routine successes.

Leading signals: catch trouble before the metric moves

Outcome metrics tell you what already happened. Leading signals warn you before they degrade, and watching them is the difference between fixing a problem in an afternoon and explaining it in a postmortem. Watch tool-call patterns: a rising rate of retries against one MCP server, or the agent suddenly choosing a different tool for a task it used to handle one way, signals drift before success rate drops. Watch turn counts: if the agent is taking more turns to resolve the same kind of task, something in the context or tool surface got harder for it to navigate. Watch escalation reasons: when the agent kicks tasks to humans, the categories of why are a live map of where it is weak.

The richest leading signal is also the lowest-tech: read transcripts. A weekly habit of reading a sample of real agent runs surfaces problems no dashboard catches — a confusing tool description the agent keeps misreading, a skill instruction it interprets too literally, an edge case quietly mishandled. Quantitative metrics tell you something is wrong; transcripts tell you what and why. The best agentic teams treat transcript review as a permanent ritual, not a debugging step they do once.

Avoiding the measurement traps

Several traps make agents look better or worse than they are. The first is survivorship in the success rate: if you only measure tasks the agent chose to attempt, and it quietly escalates the hard ones, your success rate looks fantastic while autonomy is terrible. Always measure success and autonomy together. The second is cherry-picked demos standing in for evals — the cure is a frozen eval set of real cases, including the ugly ones. The third is ignoring tail risk: averages hide the rare catastrophic action, so track safety incidents as a separate, never-averaged count.

A fourth trap is measuring the agent in isolation rather than the workflow. The honest question is whether the end-to-end process — agent plus the humans who review and escalate — is better than what you had before, in throughput, quality, and cost. An agent that scores well on its own metrics but pushes a flood of low-quality drafts onto humans has not improved the workflow. Measure the system, not just the model.

A measurement scorecard you can ship with

Put it together into a simple, recurring scorecard: task success rate against a frozen eval set, autonomy rate, cost per resolved task, and a hard count of safety incidents and near-misses, reviewed alongside a sample of transcripts every week. Set a target band for each and a rule that a regression in any one blocks scaling up until investigated. That scorecard turns "the agent feels good" into a defensible, trend-able answer to whether it is working — and gives you the evidence to expand it, tune it, or pull it back with confidence rather than hope.

Frequently asked questions

What is the single most important metric for a Claude agent?

Task success rate measured against a frozen eval set of real cases, with success defined by checkable criteria for that specific task. Without a fixed eval set, every claim of improvement is unfalsifiable. But success rate must be read alongside autonomy rate, because high success with low autonomy means humans are still doing the work.

How do I account for the higher token cost of agentic runs?

Track cost per resolved task, not cost per call. Agentic and multi-agent runs can use several times more tokens than a single model call, so the meaningful question is whether the all-in cost to resolve a task beats the alternative. An agent that succeeds but costs more than the human it replaced is not delivering value, however impressive it looks.

Why do experienced teams emphasize reading transcripts over dashboards?

Dashboards tell you that a metric moved; transcripts tell you why. Most agent problems — a misread tool description, an over-literal skill instruction, a mishandled edge case — are invisible in aggregate numbers but obvious when you read the run turn by turn. A weekly transcript-reading habit is the highest-leverage diagnostic practice in agentic engineering.

What leading signals warn me before success rate drops?

Rising tool-call retries against one MCP server, the agent taking more turns to resolve familiar tasks, shifts in which tool it chooses, and changes in escalation reasons. These move before outcome metrics degrade, so watching them lets you fix issues proactively instead of after customers feel them.

Bringing agentic AI to your phone lines

Measuring voice agents follows the same logic — success, autonomy, cost, and safety per conversation. CallSphere instruments multi-agent voice and chat assistants so you can see exactly how reliably they resolve and book work 24/7. Explore it at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.