Evals for Claude Code Agents: Gating Releases Safely

You can feel when an agent is good, and that feeling is worthless. "It seems better" is how teams ship regressions into production every week. The reason agentic systems are so easy to break silently is that they have no compile step and no red test bar — a one-line tweak to a system prompt can fix the case you were staring at while quietly breaking five cases you were not. The only honest way to know whether a Claude Code agent improved or regressed is to measure it against a fixed set of tasks with known-good expectations. That measurement loop is what evals are, and building one is the difference between an agent you hope works and an agent you can prove works.

An eval is a repeatable test that runs your agent against representative inputs and scores the results against a quality bar. This post walks through building an eval loop for Claude Code agents in a GTM context — assembling test cases, choosing graders, running regression suites, and wiring the whole thing into a release gate so nothing ships unless quality holds.

Why "it works on my prompt" is not evidence

Agent behavior is a distribution, not a value. Run the same task five times and you may get five slightly different trajectories — different tool orderings, different phrasings, occasionally a different outcome. A single successful run tells you the agent can succeed, not that it reliably does. Worse, the inputs you happen to test by hand are biased toward the cases you already understand, while production traffic is full of the messy edge cases you never thought to try. Evals fix both problems by running many representative cases, many times, and reporting an aggregate you can actually compare across versions.

The mental shift is from "did this change fix the bug" to "did this change move the score on a fixed benchmark without dropping any other score." That framing is what lets you iterate on prompts and tools quickly without flying blind, because every change gets scored against the same yardstick.

Building the eval set

Your eval set is the most valuable artifact you will build, and it should grow out of reality, not imagination. Start by mining real transcripts: the tasks your agent actually receives, including the ones it got wrong. Every production failure should become a permanent test case so that bug can never silently return — this is how an eval suite compounds in value over time. Cover the full range deliberately: the common happy path, known hard cases, adversarial inputs, and edge cases like empty results, malformed records, or ambiguous requests.

For each case, decide what "good" means before you run anything. Sometimes there is an exact expected output you can match. More often, especially in GTM work, success is fuzzier — a good lead summary, an appropriately routed ticket, a correctly enriched record. For those, you define criteria rather than a single golden string: did it pull the right fields, did it avoid hallucinating data, did it take a reasonable tool path. Writing these criteria down forces a precision about quality that vague intuition never provides.

flowchart TD
  A["Code or prompt change"] --> B["Run agent over eval set"]
  B --> C["Collect outputs & tool traces"]
  C --> D{"Grade each case"}
  D -->|Exact match| E["Programmatic check"]
  D -->|Fuzzy quality| F["LLM-as-judge rubric"]
  E --> G["Aggregate score"]
  F --> G
  G --> H{"Score >= bar & no regressions?"}
  H -->|Yes| I["Allow release"]
  H -->|No| J["Block + report failing cases"]

Choosing graders: exact, programmatic, and LLM-as-judge

Grading is where eval design lives or dies. Use the cheapest grader that captures what you care about. For structured outputs, programmatic checks are best: did the JSON parse, is the stage one of the valid enum values, does the record contain the required fields, did the agent avoid calling a forbidden tool. These are deterministic, fast, and free of judgment error, and you should push as much grading into this category as you can.

For genuinely subjective quality — tone, helpfulness, whether a summary captured the right points — use an LLM-as-judge grader: a separate model call that scores the output against a written rubric. The key to making this reliable is a specific, example-anchored rubric rather than a vague "rate this 1 to 10." Tell the judge exactly what a passing answer contains and what disqualifies one, and validate the judge itself by checking its scores against a sample you graded by hand. A judge you have not calibrated is just another opinion. Where you can, also grade the trajectory, not only the final answer — an agent that reached the right result through a wasteful or risky tool path is a latent problem even when the output looks fine.

Regression suites and the release gate

Once you have a scored eval set, the payoff is the gate. Wire the eval run into your release process so that any change to prompts, tools, or model version triggers the full suite, and the release only proceeds if the aggregate score clears your bar and no individual case regressed. That second condition matters as much as the first — a change that raises the average while breaking your three most important enterprise cases is not an improvement, and only a per-case comparison against a baseline catches it.

Set thresholds honestly. Few real agents hit 100 percent, so pick a bar that reflects acceptable production quality and a regression tolerance that reflects how costly a failure is for that task. A high-stakes write workflow should gate harder than a draft-suggestion helper. Treat the eval suite as living infrastructure: version it alongside your agent, run it in CI, and review failing cases as seriously as you would review a failing unit test, because that is exactly what they are.

Common eval pitfalls

The first pitfall is an eval set that is too small or too clean — twenty happy-path cases will pass forever while production quietly burns on the edge cases you never encoded. The second is overfitting: if you tune your prompt until it aces the eval set, you have optimized for the test, not the world, so keep a held-out slice you never tune against. The third is an uncalibrated LLM judge that drifts or grades inconsistently, silently corrupting your signal. The fourth is grading only final outputs and ignoring trajectory, which lets cost and safety problems hide behind correct-looking answers. Each of these turns a green dashboard into false confidence, which is more dangerous than no eval at all.

Frequently asked questions

How many eval cases do I need to start?

Start with twenty to fifty real cases spanning happy paths, known hard cases, and edge cases, then grow the set every time production surfaces a new failure. A modest suite that grows from real misses beats a large synthetic one that never touches reality.

When should I use an LLM-as-judge versus a programmatic check?

Use programmatic checks for anything structured or rule-based — valid fields, correct enums, forbidden tools — because they are deterministic and free. Reserve LLM-as-judge for genuinely subjective quality like tone or summary completeness, and always calibrate the judge against human-graded samples.

Should evals block a release or just report?

For anything touching real customer data or money, block. Wire the suite as a hard gate that fails the release when the score drops below the bar or any key case regresses. Reserve report-only mode for early experimentation before you trust the suite.

How do I keep from overfitting to my eval set?

Hold out a slice of cases you never tune against and check it periodically. If tuned and held-out scores diverge, you are optimizing for the test rather than the task, and it is time to refresh the set with new real-world cases.

Bringing agentic AI to your phone lines

CallSphere gates its voice and chat agents behind the same eval discipline — scored conversation suites and regression checks that must pass before any change reaches a live call. See it live at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.