Testing and evals for Claude agents: gate every release

For the first three weeks of building my app, my entire quality process was vibes. I would change a prompt, run the agent on whatever example was in front of me, decide it looked better, and ship. Then a change that obviously improved one case silently broke three others I had not thought to check. That was the day I understood why every serious team building on Claude talks about evals before they talk about prompts. Without measurement, you are not improving an agent — you are randomly walking through prompt space and hoping.

Why agents resist ordinary testing

Conventional software tests assert exact outputs: given this input, the function returns exactly that. Agents do not work that way. The same prompt can produce slightly different wording each time, and "correct" is often a range of acceptable behaviors rather than one string. On top of that, an agent's quality depends on a whole trajectory — which tools it called, in what order, with what arguments — not just the final answer. You cannot test that with a simple equality check, which is why so many people give up and fall back to vibes.

An eval is a repeatable test that scores an agent's output or behavior against a defined quality criterion, so you can measure whether a change made the agent better or worse. The shift in mindset is from "does this one example look good" to "across a representative set of cases, did my quality score go up or down." That shift is what turns agent development from guesswork into engineering.

Building a golden set you actually trust

The foundation of any eval system is a set of cases that represent the real work, each paired with a notion of what a good result looks like. I started small and honest: twenty cases drawn from real user interactions, including the awkward edge cases that had embarrassed me. For each, I wrote down what mattered — did it call the right tool, did it get the key fact right, did it stay in scope. The cases do not need to be huge in number; they need to be representative and to include the failures you most want to prevent.

Scoring is where it gets interesting. Some checks are deterministic and cheap: did the output contain the required field, did it call the database tool exactly once, did it avoid a forbidden action. Those you assert directly. Others are judgment calls — was the tone appropriate, was the explanation clear, was the answer faithful to the source. For those, an LLM-judge works well: a separate Claude call grades the output against a rubric you write. Mixing cheap deterministic checks with rubric-based judgments gives you broad coverage without hand-grading everything.

flowchart TD
  A["Proposed change"] --> B["Run agent on golden set"]
  B --> C{"Deterministic checks pass?"}
  C -->|No| D["Fail: block release"]
  C -->|Yes| E["LLM-judge scores quality rubric"]
  E --> F{"Score >= threshold & no regressions?"}
  F -->|No| D
  F -->|Yes| G["Pass: promote change"]
  G --> H["Log scores for trend tracking"]

Turning evals into a release gate

An eval that you run occasionally is a nice-to-have. An eval that runs automatically on every change and blocks the change if quality drops is a release gate, and that is where the real value lives. I wired my golden set to run whenever I touched a prompt or a tool. The rule was simple: a change ships only if deterministic checks pass and the rubric score is at least as good as the current baseline, with no regression on any case that previously passed. That gate caught the silent breakages that used to slip through, because the cases I would never have manually re-checked were checked every time.

The discipline this enforces is underrated. Once changes must pass the gate, you stop shipping plausible-looking edits and start shipping measured improvements. It also changes how you debug: when the gate fails, it tells you exactly which cases regressed, so you are fixing a specific, reproduced problem rather than chasing a vague report. The eval loop becomes both your quality bar and your debugging map.

Evaluating trajectories, not just answers

The subtlest lesson was that final-answer quality is not the whole story. An agent can arrive at a correct answer through a wasteful, fragile, or unsafe path — calling a tool five times when one would do, or taking an action it should have gated. So I added trajectory checks alongside output checks: did the agent use the expected tools, did it avoid forbidden actions, did it stay within a reasonable number of steps. These checks catch reliability and cost problems that an output-only eval would miss entirely.

This matters most as the agent gains autonomy. A read-only assistant can be judged on its answers. An agent that takes real actions must be judged on its behavior, because a bad action is bad even when the eventual answer is fine. Building trajectory awareness into evals early meant that when I later expanded the agent's powers, I already had the instrumentation to verify it was using them responsibly.

Keeping evals honest as the app evolves

Evals decay if you neglect them. Real usage drifts, new edge cases appear, and a golden set frozen on day one slowly stops representing reality. I made a habit of feeding new failures back into the set: every time a user hit a case my evals had not anticipated, that case became a permanent test. The set grows in the direction of the problems you actually have, which keeps it relevant. I also periodically audited the LLM-judge itself, spot-checking its grades against my own judgment to make sure the rubric still reflected what I cared about.

The payoff compounds. A maintained eval set is institutional memory of every mistake you have made and refuse to repeat. It lets you adopt a new model — moving to Opus 4.8 for a hard step, say — and immediately know whether quality improved, rather than guessing. By the time I shipped, my evals were the thing I trusted most, because they were the only part of the system that told me the truth whether or not I wanted to hear it.

Frequently asked questions

How many eval cases do I need to start?

Far fewer than people expect. Twenty to thirty representative cases that include your worst real failures will catch most regressions. Quality and coverage of the hard cases matter more than raw count. Grow the set over time by adding every new failure you encounter, so it tracks your actual problems.

Can an LLM-judge be trusted to grade quality?

For judgment-based criteria like tone, faithfulness, and clarity, a rubric-driven Claude judge is reliable enough to be useful, especially alongside deterministic checks. Audit it periodically against your own grades to keep it calibrated. Use deterministic assertions wherever you can, and reserve the judge for the genuinely subjective dimensions.

Should evals block releases automatically?

Yes — that is the whole point. An eval you run by hand gets skipped under deadline pressure exactly when you most need it. Wiring the gate into your change process so a quality drop blocks the change is what converts evals from a nice idea into actual protection against regressions.

Why check the agent's trajectory and not just its answer?

Because an agent can reach a correct answer through an unsafe, wasteful, or fragile path, and that matters when it takes real actions. Trajectory checks — right tools, no forbidden actions, bounded steps — catch reliability and cost problems an output-only eval cannot see. They become essential as the agent gains autonomy.

Bringing agentic AI to your phone lines

An eval loop is what lets you change a voice agent with confidence instead of fear, because you can prove quality held before anyone takes a call. CallSphere applies these same testing and evaluation patterns to its voice and chat agents, gating every release against measured quality. See it live at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.