Evals for Claude agents: measure quality, gate releases (Extending Claude Skills MCP)

Most teams ship agent changes on vibes. Someone tweaks a skill instruction, runs the agent once, the demo looks good, and it goes live. Then a user hits an edge case the demo never touched, and nobody can say whether the change made things better or worse, because there was nothing to compare against. The fix is an eval loop: a repeatable way to measure agent quality so that every change is judged by evidence, not by the last impression it left. When you extend Claude with Skills and MCP servers, evals are what turn a clever prototype into a system you can change without fear.

This post lays out how to build that loop — choosing what to measure, assembling a dataset, scoring runs including with an LLM judge, and wiring the whole thing into a release gate so quality regressions get caught before users do.

Why agent evals are harder than model evals

Evaluating a single completion is comparatively easy: feed an input, check the output against a reference. Agents break that simplicity in two ways. First, the output is a trajectory — a sequence of tool calls and reasoning steps — not just a final string, and a run can reach a correct answer through a wasteful or unsafe path you'd want to flag. Second, agent runs are non-deterministic: the same input can produce different trajectories, so a single pass tells you little. You have to measure distributions.

This means a good agent eval scores two things: outcome and process. Outcome asks whether the final result was correct or acceptable. Process asks whether the agent used the right tools, avoided loops, stayed within budget, and took no unsafe actions. A run that gets the right answer after eight redundant tool calls passed on outcome but failed on process, and you want to see both.

A working definition: an eval is a repeatable measurement of an agent's quality against a fixed dataset of cases, producing scores you can compare across versions. The word "fixed" carries weight — if your test set drifts every time you run it, you can't attribute score changes to your code changes.

Choosing metrics that mean something

Vague metrics produce vague decisions. Define concrete, measurable signals tied to what the agent is for. A task-completion rate over a realistic dataset is the backbone. Around it, layer process metrics: tool-selection accuracy, average tool calls per task, token cost per run, latency, and a count of unsafe or policy-violating actions. For tasks with a known answer, exact or semantic match works; for open-ended tasks, you'll need a rubric.

flowchart TD
  A["Proposed change to skill/tool/prompt"] --> B["Run agent over eval dataset"]
  B --> C["Score outcome: task success"]
  B --> D["Score process: tools, cost, safety"]
  C --> E{"Pass threshold & no regressions?"}
  D --> E
  E -->|Yes| F["Promote to release"]
  E -->|No| G["Block: inspect failing cases & iterate"]
  G --> A

Set explicit thresholds before you run, not after. "Task success must stay at or above the current baseline and no new safety violations" is a gate. "It seems about the same" is not. Writing the bar down in advance is what stops you from rationalizing a regression because you liked the change.

Building the dataset that anchors everything

Your eval is only as good as its cases. Start small and real: collect twenty to fifty representative tasks from actual or anticipated usage, each with inputs and a notion of what success looks like. Cover the happy path, but spend most of your effort on the edges — ambiguous requests, missing data, tools that error, inputs that previously caused loops or wrong tool calls. Every production incident should become a permanent test case so the same failure can never silently return.

Keep the dataset versioned and stable. When you add cases, note it, because your scores will shift and you need to know whether the shift came from new cases or new behavior. Over time this dataset becomes one of your most valuable assets — a precise specification of what your agent is supposed to do, written in examples rather than prose.

Scoring with code and with an LLM judge

Score deterministically wherever you can. If the task has a checkable answer — a record was created, a number matches, a required field is present — assert it in code. Deterministic checks are fast, free, and unambiguous, and you should lean on them for everything that admits a clear right answer.

For the open-ended parts — was the response helpful, did it follow the right tone, was the explanation correct — use an LLM judge: a separate Claude call that scores the output against a rubric. The judge is powerful but needs discipline. Give it a specific rubric with examples of good and bad, ask for a structured verdict rather than a vibe, and calibrate it against a sample you've labeled by hand so you trust its scores. A judge that disagrees with your human labels is itself a bug to fix before you rely on it.

Because runs are non-deterministic, execute each case several times and aggregate. Report the success rate and the variance, not a lucky single pass. A change that moves the mean success rate up and the variance down is a real improvement; a change that only got lucky once is noise.

Gating releases on the eval loop

An eval that runs only when someone remembers it is half a tool. Wire it into your release process so that any change to a skill, a tool schema, an MCP server, or the system prompt triggers the suite, and promotion is blocked unless the thresholds hold. This is the agentic analog of a test gate in CI: behavior, not just code, must pass before it ships.

The gate also creates a healthy feedback flywheel. When a release passes the suite but still fails in production, you've found a gap in your dataset — so you add the case, and the suite gets stronger. Over many cycles the eval becomes a tightening net that catches more and more of what users would otherwise catch for you, and shipping changes stops being scary.

Common eval mistakes to avoid

Three traps recur. First, scoring only the final answer and ignoring the trajectory, which lets wasteful or unsafe paths pass silently. Second, a tiny or unrepresentative dataset that gives false confidence — five happy-path cases prove almost nothing. Third, trusting an uncalibrated LLM judge whose scores nobody checked against human judgment. Avoid those three and your eval loop will tell you the truth about your agent, which is the entire point.

Frequently asked questions

How many eval cases do I need to start?

Twenty to fifty real, representative cases is enough to begin gating, provided they cover edge cases and past failures rather than only the happy path. Grow the set every time production surprises you, and prioritize coverage of failure modes over sheer volume.

Should I score the final answer or the whole trajectory?

Both. Outcome scoring tells you if the result was right; process scoring on the trajectory tells you if the agent got there efficiently and safely. A correct answer reached through loops or unsafe tool calls should still fail the process check.

How do I trust an LLM judge?

Calibrate it. Hand-label a sample of outputs, run the judge on the same sample, and confirm it agrees with your labels before relying on it. Give it a specific rubric and ask for a structured verdict, and re-check its agreement whenever you change the rubric.

How do I handle non-deterministic runs in an eval?

Run each case multiple times and aggregate into a success rate with variance rather than judging on one pass. Compare distributions across versions; a real improvement raises the mean or lowers the variance consistently, not just once.

Evals behind every conversation

Quality gates matter even more when an agent is talking to a customer live. CallSphere runs its voice and chat agents against eval suites so improvements ship on evidence, not vibes, and regressions get caught before any caller hears them. See it live at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.