Measuring success for skill-equipped AI agents

A skill-equipped Claude agent will almost always look impressive in a demo and ambiguous in production. The demo is curated; production is messy. The question that decides whether you keep investing is not "does it work?" but "is it working well enough, at scale, to be worth the trust we've given it?" Answering that requires measurement, and most teams measure the wrong things — they track how many tasks the agent attempted rather than how many it actually got right.

This post is about the metrics and signals that genuinely prove a skill-equipped agent is delivering value, and the instrumentation that makes those metrics trustworthy rather than vanity.

Outcome metrics beat activity metrics

The first discipline is to measure outcomes, not activity. "The agent handled 4,000 requests" tells you nothing about whether those requests ended well. The metrics that matter are the ones tied to the real-world result: the autonomous resolution rate (the share of tasks the agent completed correctly with no human touch), the escalation quality (when it handed off, was the handoff useful?), and the correction rate (how often did a human have to undo or fix what the agent did?).

The correction rate is the one to watch most closely, because it is the honest measure of trust. An agent with a high resolution rate but a creeping correction rate is not saving work; it is moving work downstream and adding a cleanup tax. Conversely, an agent with a modest resolution rate and a near-zero correction rate may be exactly what you want: it does less, but everything it does is right, and it routes the rest cleanly.

The eval suite as the source of truth

Production metrics tell you how the agent is doing now. An eval suite tells you whether a change will make it better or worse before you ship. A good eval suite is a set of representative tasks with known-good outcomes and graders that check the outcome rather than the wording. It is the gate every skill change passes through, and it is the single most important measurement asset you build.

flowchart TD
  A["Skill change proposed"] --> B["Run eval suite"]
  B --> C{"Resolution rate held or rose?"}
  C -->|No| D["Reject, fix skill"]
  C -->|Yes| E{"Correction rate stayed low?"}
  E -->|No| D
  E -->|Yes| F["Ship to canary"]
  F --> G["Compare live vs eval"]
  G --> H["Promote to full scope"]

The reason graders should check outcomes, not phrasing, is that agents reach correct results by many different paths. Grading on exact wording produces brittle evals that fail on harmless variation and pass on confidently wrong answers. An outcome grader asks the question that actually matters: did the refund get issued correctly, was the right slot booked, did the lookup return the safe fields? Build your evals from real production disagreements — every case where the agent and a human diverged is a candidate test — and the suite stays grounded in reality instead of in your imagination.

One discipline keeps the eval suite honest as it grows: separate the cases the agent should handle from the cases it should decline. It is tempting to fill the suite with hard problems, but an agent that correctly recognizes a case as out of scope and escalates it is succeeding, not failing. Grading must reward the right decision to not act just as much as the right action. Suites that only test the happy path produce agents that overreach, because nothing in the evals ever rewarded restraint.

Quality signals that production metrics miss

Aggregate numbers hide important behavior. Two quality signals deserve their own tracking. The first is consistency: does the agent make the same decision on equivalent inputs, or does it wobble? A skill that resolves a case correctly most of the time but occasionally veers is more dangerous than one that fails predictably, because the inconsistency erodes trust unevenly. The second is calibration: when the agent chooses to act versus escalate, is that boundary in the right place? An agent that escalates too much wastes the automation; one that escalates too little takes on work it should not.

You surface these by sampling. Pull a regular random sample of completed tasks and have a human review them blind — not just the failures, but the successes too, because a success the agent reached by a fragile path is a future failure. This human-in-the-loop review is slow and unglamorous, and it is the only way to catch the quality problems that aggregate metrics paper over.

Sampling the successes is the counterintuitive part. Reviewers naturally gravitate to failures, but a success reached by a fragile path — the agent guessed and happened to be right — is a regression waiting to happen. When a reviewer flags a correct outcome reached for the wrong reason, that case becomes a high-value eval addition, because it pins down behavior the model got away with this time and might not next time. Teams that only review failures are perpetually surprised; teams that sample successes see the fragility coming.

Cost and efficiency as first-class metrics

An agent that resolves everything but costs more than the humans it replaced is not a win. Token cost per resolved task is a real metric, and it is one teams forget until the bill arrives. Skill-equipped agents can be efficient because a focused skill keeps the model on a short, relevant path rather than reasoning from scratch every time — but only if the skills are tight. Bloated instructions and unnecessary tool calls inflate cost quietly.

Track cost per successful outcome, not cost per token, because the denominator that matters is value delivered. A change that cuts token usage but drops the resolution rate is usually a bad trade. The goal is the efficient frontier: the highest reliable resolution rate at the lowest cost per resolved task, with the correction rate held near zero.

Latency deserves a mention alongside cost, because for many agents it is the metric users actually feel. A skill that chains several unnecessary tool calls is not just expensive; it is slow, and a slow agent in a customer-facing setting erodes the experience it was meant to improve. Tracking time-to-resolution alongside cost-per-outcome catches the skills that are quietly bloated. Often the same edit that trims tokens also trims latency, because both come from cutting steps the skill did not need — a reminder that tightening skills tends to improve several metrics at once rather than trading them off.

Tying metrics to a decision

Metrics only matter if they drive a decision. The practical setup is a small dashboard with three lines — resolution rate, correction rate, and cost per resolved task — plus the eval suite as the release gate. When all three move the right way and the evals pass, you widen scope. When the correction rate creeps up, you investigate before it becomes an incident. When cost per outcome climbs, you tighten the skills.

The teams that succeed treat these numbers as a feedback loop, not a report. Every production disagreement feeds the eval set, every eval gate protects the metrics, and the metrics decide how much trust the agent earns next. That loop is what turns an impressive demo into a system you can responsibly expand.

Frequently asked questions

What is the single most honest metric for an agent?

The correction rate — how often a human has to undo or fix what the agent did. A high resolution rate can hide a rising correction tax, so tracking corrections keeps you honest about whether the agent is actually saving work or just relocating it.

How big does an eval suite need to be?

Smaller than people expect to start — a few dozen representative, outcome-graded cases catch most regressions. What matters is that the cases come from real production behavior and that the suite grows every time the agent and a human disagree.

Why grade outcomes instead of exact answers?

Because agents reach correct results by many valid paths. Grading on wording makes evals brittle and lets confidently wrong answers pass. Outcome grading asks whether the real-world result was correct, which is the only thing that matters in production.

How do we measure cost fairly?

Use cost per successful outcome, not cost per token. The value is in resolved work, so a change that lowers tokens but drops resolution is usually a loss. Aim for the highest reliable resolution rate at the lowest cost per resolved task.

Bringing agentic AI to your phone lines

CallSphere instruments its voice and chat agents on exactly these signals — resolution rate, clean escalations, and cost per booked outcome — so you can prove the automation is working. See the numbers at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.