Measuring success for Claude Agent Skills in production

It's easy to declare an Agent Skill a success because it runs. It fires, it produces output, the demo lands, everyone moves on. But "it runs" is not the same as "it works," and the gap between them is where production disappointments live. A skill can fire constantly while quietly creating rework. A multi-agent design can look impressive while burning tokens that a single skill would have saved. To know whether Skills, MCP, and subagents are actually paying off, you need metrics that measure outcomes, not activity. This post lays out the signals that matter and how to instrument them.

The framing throughout is that an agentic system succeeds when it reliably produces good outcomes at acceptable cost with shrinking human intervention. Every metric below is a different lens on that one sentence.

Why activity metrics lie

The most tempting metric is volume: how many times did the skill fire, how many tickets did it touch, how many tasks did it run. Volume feels like progress, but it tells you nothing about quality. A skill that fires a thousand times and gets half of them subtly wrong is worse than no skill, because now a human has to find and fix the wrong half. Counting runs measures effort, and effort is not the goal.

The deeper problem is that bad agentic outcomes are often invisible at the point of action. A refund issued incorrectly, a summary that drops a key fact, a deploy step skipped — these don't throw errors. They surface downstream, decoupled in time from the run that caused them. So any honest measurement program has to instrument for delayed, quality-based signals, not just the immediate "did it complete" flag. If your dashboard only shows green checkmarks, it's measuring the wrong thing.

The metrics that actually prove it works

Start with autonomy rate: the share of tasks the skill completed end to end without a human stepping in. This is the clearest signal of value, because the entire promise of Skills is removing repetitive human work. A rising autonomy rate on tasks you intended to automate means the skill is earning its place. But autonomy is only good if quality holds — which is why it never travels alone.

Pair it with rework rate: how often a human has to correct, redo, or override what the agent produced. Autonomy with high rework is a trap; you've moved work, not removed it. The combination of high autonomy and low rework is the real target. Next, escalation quality: when the skill hands a task to a human, was that the right call? A skill that escalates the genuinely ambiguous cases and confidently handles the clear ones is working; one that escalates everything is timid, and one that escalates nothing is reckless.

flowchart TD
  A["Skill completes a run"] --> B["Log outcome & tokens"]
  B --> C{"Human intervened?"}
  C -->|No| D["Count toward autonomy rate"]
  C -->|Yes, corrected| E["Count toward rework rate"]
  C -->|Yes, escalated| F{"Right call to escalate?"}
  F -->|Yes| G["Healthy escalation"]
  F -->|No| H["Over- or under-escalation signal"]
  D --> I["Quality audit on a sample"]
  I -->|Errors found| E

The diagram makes one subtle point explicit: even the runs that look fully autonomous get sampled for a quality audit, because the silent-error problem means you can't trust the completion flag alone. A run that no human touched might still be wrong; sampling is how you find those before they compound.

Cost and efficiency signals

Outcomes are half the picture; cost is the other half. The headline number is tokens per completed task, and it deserves real attention because multi-agent and subagent-heavy designs typically consume several times more tokens than a single-agent approach. That spend is justified when the task genuinely needs parallel exploration or isolated context, and wasteful when a simpler skill would have produced the same outcome. Tracking tokens per task lets you tell those cases apart instead of guessing.

Watch for the anti-pattern where someone adds subagents because it feels sophisticated and the token bill quietly triples for no quality gain. The right test is comparative: run the task both ways on a sample and see whether the extra spend buys better outcomes. If it doesn't, the simpler design wins. A good agent operator treats tokens like a budget line, with someone accountable for the trend, not as an invisible cloud cost.

Reliability and drift signals

Beyond per-run quality, you need signals that the skill stays good over time. The key one is eval pass rate on a fixed test set. An eval is a set of known inputs with known-good expected behavior; running it on a schedule turns drift from an invisible slow failure into a visible, dated event. When the world changes underneath a skill — a policy update, an API change, a new edge case — a steady eval is what tells you before customers do.

Complement evals with variance signals: is the skill behaving consistently across similar inputs, or has it become erratic? A skill that handles the same situation differently from run to run is a reliability problem even if each individual answer is defensible. And watch tool-call patterns over time — a skill suddenly calling a tool it never used, or calling one far more often than usual, is an early warning that something shifted, whether a drift, an injection, or a quietly broken upstream dependency.

Turning metrics into a feedback loop

Metrics only matter if they drive action, so wire them into a routine. A weekly review of autonomy versus rework tells you which skills to trust more and which to pull back. A drop in eval pass rate triggers an investigation before the skill is allowed to keep acting. A spike in tokens per task without a quality gain triggers a simplification. The goal is a closed loop where measurement continuously reshapes the skill library — promoting the skills that earn trust, refactoring the ones that drift, and retiring the ones that no longer pull their weight.

The teams that get the most from Skills are not the ones with the cleverest skills; they're the ones with the tightest feedback loop between what their agents do and what they measure. A mediocre skill under sharp measurement improves fast. A brilliant skill with no measurement decays silently. Instrumentation is the difference between a system you trust and a system you hope about.

Frequently asked questions

What's the single most important metric for an agent skill?

The pairing of autonomy rate and rework rate, read together. High autonomy with low rework means the skill genuinely removed work; high autonomy with high rework means it just relocated the work to whoever cleans up after it. Neither number alone tells the truth — you need both.

How do I catch errors when the skill reports success?

Sample completed runs for a quality audit and run a fixed eval set on a schedule. Agentic failures are often silent — no exception, just a wrong outcome that surfaces downstream — so you can't rely on the completion flag. Sampling and scheduled evals are how you find those errors on your terms instead of from a complaint.

When is the extra token cost of subagents justified?

When the comparative test shows it. Run the task with and without subagents on a representative sample; if the multi-agent version delivers meaningfully better outcomes, the several-times-higher token spend is earned. If outcomes match, the simpler single-skill design wins. Don't pay for sophistication that doesn't move the result.

How do I measure whether a skill is escalating correctly?

Review a sample of its escalations and its autonomous completions for correctness. Good escalation means the ambiguous cases went to humans and the clear cases were handled. Track over- and under-escalation as distinct signals: too much escalation wastes the automation, too little risks bad autonomous decisions.

Measured agents on every call

CallSphere instruments its voice and chat agents the same way — autonomy, rework, escalation quality, and cost per resolved conversation — so you can prove an assistant is working, not just that it's running. See the measured-in-production version at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.