Testing Claude Code Agents: Evals That Gate Releases

You changed one line of the system prompt to fix an edge case, shipped it, and three days later a different workflow that used to work perfectly started failing. Welcome to the central problem of agentic engineering: behavior is emergent, prompts are global, and a tweak that helps one task can silently break another. The only durable answer is the same one software learned decades ago — you don't trust a change until a test suite tells you it's safe. For agents, that test suite is an eval loop.

Evals are not optional polish. They are the instrument that lets you change a Claude Code agent with confidence instead of superstition. This post walks through what to measure, how to score non-deterministic output, and how to wire evals into a release gate so that a regression is caught by a scorecard rather than by a customer.

What you're actually measuring

An eval, at its simplest, is a task plus a way to judge the result. For agents you generally care about two layers. Outcome evals ask: did the agent achieve the goal? Did the test suite end green, did the file get created with the right contents, did the extracted data match the expected schema? These are the evals that matter most because they map to what users want. Trajectory evals ask a different question: did the agent get there sensibly — without looping, without calling a destructive tool it didn't need, without burning ten times the necessary tokens? A run can reach the right outcome through a wasteful or risky path, and you want to catch that too.

Build your eval set from real cases. The best test cases are the failures you've already seen — every time the agent does something wrong in production, capture that scenario and add it to the suite. Over time this becomes a regression net that encodes everything the agent has ever gotten wrong, which is exactly what you want guarding the door before a release.

Scoring non-deterministic output

The hard part of agentic evals is that the output isn't a single deterministic string. The agent might write the same function two valid ways, or phrase a summary differently each run. You need scoring that tolerates surface variation while still catching real regressions. Three approaches cover most needs, and mature suites combine all three.

The most reliable is a programmatic check: run the code, assert the test passes, validate the JSON against a schema, diff against an expected artifact. When a task has an objective success condition, encode it as an assertion and you get a fast, unambiguous signal. The second is an LLM judge — a separate model call scoring the output against a rubric — for qualities you can't assert mechanically, like whether an explanation is clear or a response stayed on-topic. The third is human review on a sampled subset, which stays the ground truth you calibrate the automated judges against.

flowchart TD
  A["Prompt / config change"] --> B["Run eval suite\non fixed task set"]
  B --> C["Programmatic checks"]
  B --> D["LLM judge on rubric"]
  C --> E["Aggregate scorecard"]
  D --> E
  E --> F{"Score >= threshold\n& no regressions?"}
  F -->|Yes| G["Allow release"]
  F -->|No| H["Block & show diffs"]

An LLM judge done right

An LLM judge is powerful and easy to get wrong. The failure mode is a vague rubric that produces noisy, drifting scores you can't trust. Fix that by making the rubric concrete: instead of "rate quality 1-10," ask specific yes/no questions — "does the response answer the user's actual question," "does it avoid claiming facts not in the provided context," "is the code free of the bug described in the task." Binary, well-defined criteria are far more stable than a fuzzy numeric scale.

Validate the judge before you rely on it. Score a batch of outputs with both the judge and a human and check that they agree; if they diverge, the rubric is the problem, not the agent. Use a capable model for judging — a stronger Claude as the judge over a weaker model's output is a common and effective setup — and keep the judge's prompt under version control alongside the agent's, because a change to how you measure is as consequential as a change to what you're measuring.

Gating releases on the scorecard

Evals only protect you if they sit between a change and production. Wire the suite into CI so that every prompt edit, tool change, model swap, or skill update triggers a run against the full task set, and the result is a scorecard: pass rate per task, aggregate score, token cost, and a diff against the previous baseline. The release gate is a simple rule — the aggregate must clear a threshold and no previously passing task may regress.

That second clause matters more than the first. A small overall improvement that quietly breaks two formerly green tasks is the exact failure that hand-testing misses, and a per-task regression check catches it cold. Treat any new regression as a blocker, investigate it, and either fix the change or consciously accept the trade-off — but never let it slip through unseen. This is also how you safely adopt a new model: run the candidate against the same suite and compare scorecards before switching anything in production.

Keeping evals honest over time

An eval suite is software and it rots like software. As the agent's job evolves, stale tests start asserting behavior you no longer want, and the suite either blocks good changes or, worse, passes while missing the cases that now matter. Budget time to prune dead tests, add cases for every new failure, and periodically re-check that your LLM judges still agree with human reviewers.

Beware of overfitting, too. If you tune prompts until they ace a fixed eval set, you may be optimizing for the test rather than the task. Keep a held-out set of cases the agent's development never sees directly, and check performance there before a release. A definition to anchor on: an eval loop is the repeatable cycle of running an agent against a fixed set of representative tasks, scoring the results against objective and rubric-based criteria, and using that scorecard to decide whether a change ships. Build that loop once and every future change becomes a measured decision instead of a gamble.

Frequently asked questions

How do you test something as non-deterministic as an agent?

Score with criteria that tolerate surface variation. Use programmatic checks where there's an objective success condition — run the code, assert tests pass, validate against a schema — and an LLM judge with a concrete yes/no rubric for qualities you can't assert mechanically. Sample human review to keep the automated scores calibrated.

What should go in a Claude Code eval set?

Real cases, especially past failures. Every time the agent gets something wrong in production, capture that scenario as a test. Over time the suite becomes a regression net encoding everything the agent has ever gotten wrong, which is exactly what you want guarding releases. Include both outcome and trajectory checks.

How do I gate a release on evals?

Run the full suite in CI on every prompt, tool, model, or skill change and produce a scorecard. The gate is twofold: the aggregate score must clear a threshold, and no previously passing task may regress. The regression clause catches the silent breakage that a small overall improvement can hide.

Can I trust an LLM judge to score my agent?

Only after you validate it. Use concrete binary rubric questions instead of a fuzzy 1-10 scale, then check the judge's scores against human reviewers on a sample. If they diverge, fix the rubric. Keep the judge prompt in version control, since how you measure is as consequential as what you measure.

Bringing agentic AI to your phone lines

This same eval-gated discipline is how CallSphere keeps its voice and chat agents reliable in production — assistants that answer every call and message, use tools mid-conversation, and book work 24/7, with every change measured before it ships. See it live at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.