Testing Claude agents: evals that gate every release

Here is the uncomfortable truth about agentic systems: you cannot tell whether a change to a Skill made things better or worse by reading the diff. Tweak one instruction, sharpen one tool description, swap a model, and the agent's behavior shifts in ways that ripple across dozens of multi-step runs. Without measurement, you are flying blind — every "improvement" is a guess, and regressions slip into production unnoticed until a customer hits one. The teams who ship reliable Claude agents in 2026 all share one habit: they built an eval loop early and they let it gate every release. This post is how to build that loop.

Let us define the thing precisely. An eval is a repeatable test that runs your agent or Skill against a fixed set of inputs and scores the outputs against an expected standard, producing a number you can track over time. The number is the point. "It seems better" is not a release criterion; "the pass rate went from 82% to 91% with no regression on the safety suite" is. Evals turn agent quality from a vibe into an engineering metric.

What to actually measure

Agents have more failure dimensions than a single chat completion, so measure several. Task success is the headline: did the agent achieve the goal? But also measure the trajectory — did it call the right tools in a sensible order, or did it stumble to the answer through ten wrong turns? A run that succeeds by luck after looping is fragile and will fail tomorrow. Measure cost and latency too, because a Skill that doubles accuracy while tripling tokens may not be a win.

Build your eval set from reality. Mine real transcripts for the cases your agent actually faces, and over-sample the hard ones: ambiguous requests, missing data, adversarial inputs, edge cases that previously broke. Every production bug should graduate into a permanent eval case so it can never silently return. A good eval set is small enough to run often and representative enough that passing it means something.

Choosing your graders

How you score matters as much as what you score. For tasks with a deterministic right answer — a parsed value, a chosen tool, a final id — use exact or programmatic checks. They are cheap, fast, and unambiguous, so prefer them wherever the output structure allows. Assert on the tool sequence, assert on the final structured result, and you get a reliable signal with no model in the loop.

For open-ended outputs — a summary, an explanation, a customer reply — use an LLM judge: a separate Claude call given the input, the output, and a rubric, asked to score against explicit criteria. The discipline that makes judges trustworthy is a sharp rubric. "Is this good?" produces noise; "Does the reply answer the question, cite the right record, and avoid promising anything not in the data? Score each 0 or 1" produces a signal you can act on. Validate your judge against a sample of human labels so you trust its agreement before relying on it.

flowchart TD
  A["Change a Skill or prompt"] --> B["Run eval set"]
  B --> C{"Grader type?"}
  C -->|Deterministic| D["Programmatic check"]
  C -->|Open-ended| E["LLM judge + rubric"]
  D --> F["Aggregate scores"]
  E --> F
  F --> G{"Pass gate & no regression?"}
  G -->|Yes| H["Ship release"]
  G -->|No| I["Block, inspect failures, fix"]
  I --> A

Gating releases with the eval loop

An eval that runs only when you remember it is theater. Wire it into your release process so no Skill or prompt change ships without passing. Define gates explicitly: an overall pass-rate threshold, a hard zero-tolerance on a safety subset, and a no-regression rule against the last known-good baseline. If a change improves average success but breaks two previously passing cases, the gate should stop it until you understand why. Regressions are how trust erodes.

Run the full suite in CI on every change to an agent's Skills, tools, or prompts, and run a fast smoke subset on each commit for quick feedback. Treat the eval suite like a test suite, because that is exactly what it is — the unit and integration tests of an agentic system. The moment evals become a required check rather than an optional courtesy, your agent's quality stops drifting and starts climbing.

Handling nondeterminism

Agents are stochastic, so a single run is a noisy sample. A case that passes once may fail the next time on the same input. The fix is to run each eval case several times and report a pass rate rather than a binary, so you measure reliability, not a lucky draw. For the cases that must never fail, require a high pass rate across many runs before you call them green.

Hold sampling settings steady across eval runs so you are comparing like with like — changing temperature between baseline and candidate confounds the result. And watch variance itself as a metric: a change that keeps the same average but widens the spread has made your agent less predictable, which in production feels like a regression even if the mean says otherwise.

Closing the loop over time

The eval set is a living asset. As your agent meets new inputs in production, harvest the surprising and the broken into new cases. As you fix bugs, lock them in. As you add Skills, add evals that exercise them and confirm they do not degrade existing behavior. Over months, this compounding suite becomes your most valuable artifact — it encodes everything you have learned about what "working" means for your specific system, and it lets a new engineer change a Skill on day one without fear, because the gate will catch them if they break something.

Resist the urge to chase a perfect score on a frozen set; that just teaches you to overfit. Keep refreshing the inputs, keep raising the bar on the hard cases, and treat the eval pass rate as a north star you steer toward rather than a box you tick. The goal is not a green dashboard; it is an agent you can change quickly and ship confidently because measurement, not hope, tells you it works.

Frequently asked questions

When should I use an LLM judge versus a programmatic check?

Use programmatic checks for outputs with a deterministic right answer — parsed values, chosen tools, final ids — because they are cheap and unambiguous. Use an LLM judge with a sharp rubric for open-ended outputs like summaries and replies, and validate it against human labels first.

How big should my eval set be?

Small enough to run often, representative enough that passing it means something. Start with a few dozen real cases weighted toward hard and previously broken inputs, and grow it by graduating every production bug into a permanent case.

How do I handle the fact that agents are nondeterministic?

Run each case multiple times and report a pass rate rather than a single pass/fail, hold sampling settings constant across runs, and track variance as its own signal — a wider spread at the same average is a reliability regression.

What gates should block a release?

An overall pass-rate threshold, zero tolerance on a safety subset, and a no-regression rule against the last known-good baseline. A change that lifts the average but breaks previously passing cases should be blocked until explained.

Bringing agentic AI to your phone lines

CallSphere runs this exact eval discipline behind voice and chat agents — measured, gated, and regression-tested — so they answer every call and message and book work 24/7 with quality you can trust. See it live at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.