Testing & Evals for Claude Agents: Gate Releases Safely

You can't ship an agent you can't measure. The most dangerous moment in building a Claude orchestration system is the point where it works in the demo — the happy path looks brilliant — and the team starts changing prompts and tools on vibes. One tweak fixes the case you were staring at and silently breaks three you weren't. Without an evaluation loop, every change is a coin flip and quality drifts in whatever direction the last bug happened to push it. This post is about building the loop that replaces vibes with numbers: how to measure agent quality, where LLM judges help and where they mislead, and how to make a passing eval the gate that every release has to clear.

What you're actually measuring

An eval is a repeatable test that scores your system's output against a defined expectation. For agents, "output" has two layers, and good evals look at both. The first is the outcome: did the run produce the right final result? For a refund agent, was the refund the correct amount? For a research agent, did the summary contain the key facts and avoid false ones? The second is the trajectory: how did it get there? An agent that reaches the right answer after fourteen flailing tool calls and two loops is a latent failure even when the outcome is correct, because the next slightly different input will tip it over.

Score both. Outcome evals catch wrong answers; trajectory evals catch fragility — unnecessary turns, wrong tool selections, retries, and excess token spend. Together they tell you not just whether the agent is right today but whether it's robust enough to stay right tomorrow.

Building the eval set

An eval is only as good as its cases. Build the set from three sources. First, hand-write a core of canonical tasks that represent the work the agent must do, each with a clear expected outcome. Second, and most valuable, mine production: every real failure you debug becomes a permanent eval case, so the system can never regress on a bug you've already paid to find once. Third, deliberately add adversarial and edge cases — ambiguous requests, missing data, hostile input — because those are where agents break and where the happy-path demo told you nothing.

Keep each case self-contained and deterministic to set up: fixed inputs, mocked or recorded tool responses where you need reproducibility, and an explicit definition of "pass." The goal is a suite you can run on every change in minutes and trust the result.

flowchart TD
  A["Prompt or tool change"] --> B["Run eval suite"]
  B --> C["Outcome scores"]
  B --> D["Trajectory scores"]
  C --> E{"Pass rate >= threshold?"}
  D --> E
  E -->|No| F["Block release & show failing cases"]
  E -->|Yes| G{"Any regression vs baseline?"}
  G -->|Yes| F
  G -->|No| H["Promote to production"]
  F --> A

Using an LLM judge without fooling yourself

Many agent outputs — a summary, an explanation, a customer reply — have no single correct string to assert against, so you grade them with another model. An LLM-as-judge is a model you prompt to score an output against a rubric. Used well, it scales human judgment to thousands of cases. Used carelessly, it gives you confident, meaningless numbers.

Three rules keep judges honest. Make the rubric concrete and binary wherever possible — "does the reply state the correct refund amount: yes/no" beats "rate helpfulness 1–10," which different runs score differently for no reason. Calibrate the judge against human labels on a sample; if the judge and your humans disagree often, the judge prompt is wrong and its scores are noise until you fix it. And don't let the same model family judge its own work without spot-checks, because shared blind spots mean it can rate a subtly wrong answer as perfect. For anything strictly checkable — a number, a tool that should or shouldn't have been called, a required field — use a deterministic assertion, not a judge. Reserve the judge for genuinely subjective quality.

Gating releases on the eval

The loop only protects you if it has teeth. Wire the eval suite into your deployment so that a prompt change, a tool description edit, or a model swap cannot reach production unless it clears two bars: an absolute pass-rate threshold and a no-regression check against the current baseline. The regression check matters as much as the threshold, because the failure that hurts is the one where you fixed a new case and broke an old one — an absolute score can stay flat while specific important cases flip. Show the engineer exactly which cases failed, with the trajectory, so a red build is a debugging lead rather than a wall.

This is the same discipline that test suites brought to ordinary software, adapted to non-determinism. Because agent runs vary, run flaky-prone cases a few times and require a pass rate rather than a single pass, and track scores over time so slow drift becomes visible before it becomes an incident.

What to watch in production

Offline evals can't cover every input real users send, so close the loop with online signals. Sample real runs and score them with the same judges and assertions you use offline. Watch the trajectory metrics — average turns per run, tool-call counts, loop incidents, token spend — and treat a creeping average as an early warning. And make it trivial to promote any production surprise into the offline eval set, so the suite gets stronger every week and your gate keeps getting harder to fool.

Frequently asked questions

How many eval cases do I need to start?

Fewer than you think — a few dozen well-chosen cases covering the core tasks plus your worst known failures already catches most regressions. The set should grow continuously from production, not be perfect on day one. Coverage of real failure modes matters far more than raw count.

When should I trust an LLM judge versus a hard assertion?

Use a deterministic assertion for anything objectively checkable — a number, a required field, whether a specific tool was called. Reserve the LLM judge for subjective quality like tone, completeness, or coherence, and only after you've calibrated it against human labels so you know its scores track reality.

How do I handle non-determinism in evals?

Run sensitive cases multiple times and require a pass rate rather than a single pass, fix tool responses where reproducibility matters, and gate on both an absolute threshold and a no-regression comparison. Tracking scores over time turns inherent variance into a trend you can actually reason about.

Evals behind your phone-line agents

CallSphere runs this same eval-gated loop — outcome and trajectory scoring, calibrated judges, no-regression release gates — for voice and chat agents that answer every call and message and book work 24/7, so quality is measured, not hoped for. See the result live at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.