Evals for Claude Agents: Gating Releases With a Loop

Here is the question that separates teams who ship reliable agents from teams who ship anxiety: when you change a skill, how do you know you made the agent better and not worse? Without a way to answer that, every edit is a gamble. You tweak an instruction to fix one user's complaint, deploy, and silently break three other workflows you forgot existed. The cure is the same one that made software testing routine — a repeatable evaluation loop — adapted to the fact that agents are non-deterministic and judged on quality rather than exact equality. This post is about building that loop for agents made with Claude Agent Skills.

An evaluation, or eval, is a structured test that measures how well an agent performs a defined task against expected outcomes. Unlike a unit test that asserts a single correct string, an agent eval has to tolerate that two good answers can be worded differently and still both be right. That tolerance is what makes agent evals harder than ordinary tests — and what makes a disciplined approach so valuable, because the teams that get it right can iterate fearlessly while everyone else edits and prays.

Start with a real test set, not a hypothetical one

The foundation is a collection of representative cases: realistic inputs paired with what a good outcome looks like. The fastest way to build a useful set is to mine real usage. Every interesting run — especially every failure your users report — becomes a case. Capture the input, the environment state, and a note on what the agent should have done. A few dozen well-chosen real cases beat hundreds of synthetic ones, because they reflect the messy distribution your agent actually faces rather than the clean one you imagined.

Deliberately include the hard cases: ambiguous requests, inputs that should trigger a clarifying question, situations where the correct move is to refuse or escalate. Agents fail most often at the edges, so an eval set that is all happy-path will pass right up until production embarrasses you. The set is a living asset — every new failure mode you discover gets added, so the agent can never regress on a bug you have already seen.

flowchart TD
  A["Skill change proposed"] --> B["Run agent across eval set"]
  B --> C["Score each case"]
  C --> D{"Pass rate above bar?"}
  D -->|Yes| E["Allow release"]
  D -->|No| F["Block & show failing cases"]
  F --> G["Fix skill or add example"]
  G --> B

Choose a scoring method that fits the task

How you score depends on what you are measuring. Some outputs are checkable deterministically — did the agent call the right tool, did the JSON validate, did it extract the correct order number? Use exact or programmatic checks wherever you can, because they are fast, free, and unambiguous. Reserve subjective judgment for the parts that genuinely need it: tone, completeness, whether the reasoning was sound.

For those subjective dimensions, LLM-as-judge is the practical workhorse: a separate model call scores the output against a written rubric. The rubric is everything. "Rate this answer 1-10" produces noise; "Score 1 if the answer resolves the user's question without inventing facts, 0 otherwise, and cite the sentence that justifies your score" produces signal. Spot-check the judge against your own human ratings periodically — if the judge and a human disagree often, the rubric is too vague and needs sharpening before you trust it to gate anything.

Wire the eval loop into release gating

An eval set that you run manually once a month is barely better than none. The payoff comes from making the eval a gate. Run the suite automatically on every meaningful change to a skill or prompt, in CI, and block the change if the pass rate drops below your bar. Now a regression is caught the moment it is introduced, by the person who introduced it, while the context is fresh — not three weeks later in a production incident.

Set the bar deliberately and report the deltas, not just a green checkmark. Knowing the suite went from 94% to 91% and seeing exactly which three cases newly failed is far more actionable than a binary pass/fail. Track the score over time the way you track test coverage; a slow decline across releases is a signal that quality is eroding even when no single change tripped the gate.

Account for non-determinism without going crazy

Agents do not produce the same output twice, which unsettles engineers used to deterministic tests. The answer is not to chase determinism but to design evals that tolerate variance. Run each case more than once and look at the pass rate across runs rather than treating a single execution as ground truth; a case that passes four times out of five tells you something a single run cannot. Set the gate on aggregate behavior — "at least 90% of cases pass, and no critical case ever fails" — rather than demanding a flawless single pass that a non-deterministic system will never reliably deliver.

Separate the cases by stakes while you are at it. A handful of cases are critical: the agent must never leak data, must never take a destructive action it was not asked to, must always escalate a genuine emergency. Treat those as hard gates that block on a single failure, regardless of the overall pass rate. The long tail of ordinary quality cases can tolerate the occasional miss and be judged in aggregate. Mixing the two — applying the same lenient aggregate bar to a safety-critical case — is how a serious failure slips through a green dashboard.

Evaluate the path, not only the final answer

Agents are sequential, so a correct final answer can hide a broken process — the agent stumbled through six wrong tool calls before luckily landing somewhere acceptable. Score the trajectory as well as the outcome: did it pick the right tools, in a sensible order, without wasteful loops? An agent that gets the right answer inefficiently is a cost and latency problem waiting to surface at scale, and trajectory scoring is how you catch it before the invoice does.

This is also where evals connect back to everything else you measure. The same captured runs that feed your debugging fixtures and your cost dashboards feed your eval set. One stream of real production traffic, replayed and scored, tells you simultaneously whether the agent is correct, efficient, and safe. Building that single replayable corpus is the highest-leverage infrastructure investment an agent team can make.

One caution as the suite grows: guard against overfitting to it. If you only ever tune the agent against your eval cases, you can drift into a skill that aces the test set and quietly degrades on the live traffic the set fails to represent. Keep refreshing the corpus with new production samples, hold a portion back as a validation set you do not optimize against directly, and periodically have a human read a random sample of recent live runs. The eval suite is a proxy for reality, not a substitute for it, and the moment it stops reflecting real usage it starts giving you false confidence.

Frequently asked questions

How big does an eval set need to be?

Start small and real — a few dozen cases drawn from actual usage and known failures beat a large synthetic set. Grow it every time you discover a new failure mode, so coverage tracks the bugs that actually occur. Quality and representativeness matter far more than raw count.

Is LLM-as-judge reliable enough to gate releases?

Yes, when the rubric is specific and you periodically validate the judge against human ratings. Use deterministic checks for anything programmatically verifiable and reserve the model judge for subjective qualities like tone and completeness. A vague rubric makes the judge noisy; a precise one makes it a dependable gate.

Should evals run on every change?

Run them automatically on every meaningful change to a skill or prompt, in CI, and block releases that drop below your pass bar. Catching a regression at the moment of introduction, with the failing cases shown, is dramatically cheaper than discovering it in production weeks later.

Why evaluate the trajectory and not just the answer?

Because a correct answer can come from a broken, wasteful process — six bad tool calls that happened to end well. Scoring the path catches inefficiency, unnecessary loops, and wrong tool choices that the final answer hides, which directly affects cost, latency, and reliability at scale.

Bringing agentic AI to your phone lines

CallSphere gates its voice and chat agents behind exactly this kind of eval loop — replaying real conversations, scoring them against rubrics, and blocking any change that regresses quality before it reaches a live call. See it live at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.