How to Measure Agent Skill Success: Metrics That Matter

"It works on my machine" is even more dangerous for Agent Skills than for code, because skills are non-deterministic — the same prompt can produce different behavior on different runs. A skill that looked flawless in a demo can be quietly failing one in six times in production, and without the right metrics you'll never know. Measuring an Agent Skill is not about a single accuracy number; it's about a small dashboard of signals that, together, tell you whether the skill triggers correctly, does the right thing, does it consistently, and does it affordably. This post defines those metrics precisely and shows how to compute them with skill-creator.

Key takeaways

• Measure four families: trigger accuracy, outcome quality, consistency (variance), and cost.
• Trigger accuracy needs both recall (fires when it should) and false-trigger rate (stays quiet when it shouldn't).
• Outcome quality requires a rubric or grader, not a gut check; define "good" before you measure.
• Variance across multiple runs is the metric most teams skip and the one that predicts production pain.
• Always record a baseline so every future edit is judged against a known number, not a feeling.

What does "working" actually mean for a skill?

A useful definition: a skill is working when it loads on the intended inputs, stays dormant on everything else, produces outputs that meet a defined quality bar, does so consistently across repeated runs, and does it at acceptable token cost. Each clause maps to a metric, and a skill can ace one while failing another. A skill with perfect outputs that only triggers half the time is not working. Neither is one that triggers perfectly but produces a different answer every run. You need the full set.

How do the metrics connect to an eval loop?

The diagram shows how raw eval runs become the four signals and how they gate a release.

flowchart TD
  A["Labeled eval set"] --> B["skill-creator runs N times per case"]
  B --> C["Trigger log: fired vs expected"]
  B --> D["Outputs graded by rubric"]
  B --> E["Token & latency captured"]
  C --> F["Recall & false-trigger rate"]
  D --> G["Quality score + variance"]
  E --> H["Cost per task"]
  F --> I{"All four meet bar?"}
  G --> I
  H --> I
  I -->|Yes| J["Ship & record baseline"]

The key idea is that one set of runs produces all four signals at once. You don't run separate experiments for trigger accuracy and cost — you instrument a single eval pass and derive every metric from it.

Defining each metric concretely

Trigger recall is the fraction of should-trigger cases where the skill actually loaded. False-trigger rate is the fraction of should-not cases where it loaded anyway. Outcome quality is the share of triggered runs whose output passes your rubric — exactness, grounding, length, tone. Variance is how much quality swings across the N runs of the same case; low variance means predictable behavior. Cost per task is mean tokens (and latency) for a completed run, which matters because multi-step skills can quietly become expensive.

A subtlety worth internalizing: these metrics interact. Tightening a description to cut the false-trigger rate can also drop recall if you over-constrain it, so you tune them as a pair rather than in isolation. Likewise, pushing quality up by adding more instruction and resources tends to raise token cost, so a quality gain that doubles cost may not be worth shipping. The dashboard exists precisely so you can see these tradeoffs at a glance instead of optimizing one number into a corner. Treat any single metric moving sharply while the others stay flat as a signal to look closer, not to celebrate.

A rubric grader you can paste into your eval

Outcome quality only means something if "good" is written down. The following rubric, applied by an LLM grader or a human, turns a vague "the reply looks fine" into a repeatable score. It is small on purpose — narrow rubrics are more reliable than sprawling ones.

{
  "rubric": "ticket-triage output",
  "checks": [
    { "id": "summary_len",  "pass_if": "summary is exactly 3 sentences" },
    { "id": "draft_present", "pass_if": "a reply draft exists" },
    { "id": "draft_len",    "pass_if": "reply is under 120 words" },
    { "id": "grounded",     "pass_if": "every claim traces to policy.md" },
    { "id": "no_invention", "pass_if": "no policy not in policy.md" }
  ],
  "score": "fraction of checks passed, averaged over all runs"
}

Run this across every run of every case and you get both a mean quality score and its variance — the two numbers that decide whether the skill ships.

Common pitfalls in measuring skills

• Reporting the best run. A single great run is noise. Always report the mean and the spread across many runs.
• Measuring quality without measuring triggers. A skill can score 95% on outputs and still be useless if it only fires 60% of the time. Track both.
• No false-trigger denominator. Recall alone is gameable — a skill that fires on everything has perfect recall and terrible precision. Always include should-not cases.
• Vibe-based quality grading. "Looks good" doesn't survive a teammate disagreeing. Write a rubric and apply it consistently.
• Ignoring cost until the bill arrives. Multi-step and multi-agent skills can use several times more tokens than you expect; capture cost in the same eval pass.

Stand up skill metrics in five steps

1. Build a labeled eval set with both should-trigger and should-not cases.
2. Write a small rubric that defines a passing output.
3. Run the eval with several runs per case using skill-creator, capturing triggers, graded outputs, and tokens.
4. Compute recall, false-trigger rate, mean quality, variance, and cost per task.
5. Set a release bar on each, record the baseline, and re-run on every change.

The four metric families at a glance

Metric	What it answers	Healthy signal
Trigger recall	Does it fire when needed?	High and stable
False-trigger rate	Does it stay quiet otherwise?	Low, near zero
Outcome quality	Is the result good?	High mean by rubric
Variance	Is it consistent?	Low spread across runs
Cost per task	Is it affordable?	Within token budget

Frequently asked questions

How many runs per case are enough?

Enough to see the variance — often a handful for a stable skill, more for one near its quality bar. If your numbers move a lot between runs, run more.

Can I use Claude itself to grade outputs?

Yes, an LLM grader against a tight rubric is practical and scalable. Spot-check its judgments against human grades periodically to keep it honest.

Which metric is the best early-warning sign?

Variance. A skill whose quality swings between runs is telling you it's brittle long before the mean drops.

Do I need cost metrics for simple skills?

Less so for a one-shot skill, but the moment a skill chains steps or spawns subagents, capture cost — it can rise several-fold quickly.

Measured agents on your phone lines

CallSphere instruments voice and chat agents with these exact signals — trigger accuracy, grounded quality, consistency, and cost — so every call is handled well and you can prove it. See the metrics in action at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.