How to measure if your Claude agent is actually working (Building Effective AI Agents)

An agent demo is the easiest thing in the world to fake yourself out with. It works three times in a row in the meeting, everyone nods, and it ships. Two weeks later the support queue is full of edge cases it never saw. The gap between "looked good in the demo" and "is actually working" is measurement — and measuring agents is genuinely different from measuring a service, because the thing you care about isn't latency or uptime, it's whether the agent produced the right outcome. This post lays out the signals that actually prove an agent works.

Key takeaways

• The north-star metric is task success rate — did the agent achieve the real outcome — not token counts or response time.
• Maintain an eval pass rate on a frozen labeled set as your pre-production and regression signal.
• Track cost per successful outcome, not cost per call; a cheap agent that fails is expensive.
• Human-handoff rate and intervention rate reveal where the agent quietly can't cope.
• Watch leading signals (tool-error rate, loop depth, escalations) to catch decay before outcomes drop.

Why standard metrics mislead

If you instrument an agent like a microservice, you'll measure latency, error rate, and throughput — and you'll learn almost nothing about whether it's helping. An agent can respond in 800ms, return HTTP 200 every time, and be confidently wrong on a third of tasks. The metrics that matter are outcome metrics, and they require you to define what a successful outcome is for your specific task. That definition is the hard, valuable work; the dashboards are easy once you have it.

There are two families of metrics and you need both. Offline metrics run against a frozen labeled eval set and tell you whether a change is safe to ship. Online metrics run against live traffic and tell you whether the agent is actually delivering. Teams that only do one fly blind in the other direction.

What signals should you actually track?

It helps to see how a single agent run feeds your metrics. Every run emits signals at three layers — outcome, behavior, and cost — and each layer answers a different question.

flowchart TD
  A["Agent run completes"] --> B["Outcome layer: task succeeded?"]
  A --> C["Behavior layer: tools, loops, handoff?"]
  A --> D["Cost layer: tokens & latency"]
  B --> E["Task success rate"]
  C --> F["Handoff & tool-error rate"]
  D --> G["Cost per successful outcome"]
  E --> H{"Trending down?"}
  F --> H
  H -->|Yes| I["Investigate & add eval case"]
  H -->|No| J["Healthy — keep monitoring"]

The outcome layer is your north star. The behavior layer explains why outcomes move. The cost layer keeps you honest about whether the win is economical. When the outcome trend dips, the behavior signals usually told you first.

A concrete example: logging outcomes for measurement

You can't measure success rate if you never recorded what happened. The pattern is to emit a structured record at the end of every agent run capturing the outcome and the behavior signals together.

def log_run(trace_id, task, result, usage):
    record = {
        "trace_id": trace_id,
        "task": task,
        "succeeded": result["resolved"] and not result["reopened"],
        "handed_off": result["escalated_to_human"],
        "tool_errors": result["tool_error_count"],
        "loop_depth": result["steps"],
        "input_tokens": usage.input_tokens,
        "output_tokens": usage.output_tokens,
    }
    metrics.emit(record)

# success rate = succeeded / (total - handed_off)
# cost per success = sum(token_cost) / sum(succeeded)

Two definitions in the comments do most of the work. Success rate excludes clean handoffs from the denominator — an agent that correctly escalates a hard case shouldn't be punished. Cost per success divides total spend by successful outcomes, so an agent that's cheap but failing shows up as expensive, which is the truth.

Common pitfalls in measuring agents

• Measuring activity, not outcomes. "Handled 10,000 tickets" says nothing about how many it handled correctly. Fix: define and track task success, not volume.
• Cost per call instead of cost per outcome. A failing agent that's cheap per call quietly costs more once humans clean up. Fix: divide spend by successful outcomes.
• Ignoring handoff rate. A rising handoff rate is the agent telling you it's hitting its limits. Fix: chart it as a first-class metric and investigate spikes.
• Only offline or only online. Evals without production telemetry miss real drift; telemetry without evals can't gate releases. Fix: run both.
• No leading indicators. Waiting for outcome metrics to drop means waiting for users to be hurt. Fix: alert on tool-error rate and loop depth, which move first.

Stand up agent metrics in 6 steps

1. Write a one-sentence definition of "successful outcome" for your specific task.
2. Emit a structured run record with outcome, handoff, tool errors, loop depth, and tokens.
3. Compute task success rate (excluding clean handoffs) and cost per successful outcome.
4. Freeze a labeled eval set and track pass rate as your pre-ship gate.
5. Add leading-indicator alerts on tool-error rate and loop depth.
6. Every production surprise becomes a new eval case so the gold set grows with reality.

Which metric answers which question

Metric	Answers	Type
Task success rate	Is it actually working?	Online north star
Eval pass rate	Is this change safe to ship?	Offline gate
Cost per successful outcome	Is the win economical?	Online economics
Human-handoff rate	Where can't it cope?	Online behavior
Tool-error / loop depth	Is decay coming?	Leading indicator

Task success rate is the share of agent runs that achieve the intended real-world outcome — a resolved ticket, a booked appointment, a correct extraction — and it is the only metric that directly answers whether an agent is working, which is why every other signal exists to explain or protect it. Optimize the leading indicators and the eval pass rate, and the north star tends to take care of itself.

Frequently asked questions

Isn't a high response rate good enough?

No. An agent can respond to everything and be wrong on a quarter of it. Responsiveness is table stakes; correctness of outcome is the metric that matters.

How do I measure success when there's no clean ground truth?

Use proxies: did the user re-open the ticket, did they have to repeat themselves, did a human override the agent. Combine several weak signals into a success definition, and validate it against a labeled sample.

Should a handoff count as a failure?

No — a correct, timely handoff is a success of judgment. Exclude clean handoffs from the success denominator and only count it against the agent when it escalates something it should have handled.

What's the fastest signal that an agent is degrading?

Behavioral leading indicators — rising tool-error rate and increasing loop depth — usually move before outcome metrics, giving you time to investigate before users feel it.

Bringing measured agents to your phone lines

CallSphere instruments voice and chat agents on exactly these signals — task success, handoff rate, and cost per booked outcome — so you can see they work, not just hope they do. Explore the live metrics at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.