Evals for Claude agents: measure quality and gate releases

Every team building Claude agents eventually hits the same wall. The agent works in the demo, ships, and then a small prompt tweak quietly breaks a category of tasks nobody noticed until a customer did. The team rolls back, adds the case to a mental checklist, and the cycle repeats. The way out of that cycle is evals - a systematic way to measure whether your agent is actually good, so that you can change it with confidence instead of crossing your fingers. Without evals, every change to a prompt, a tool, or a model is a guess. With them, it is a measurement.

This article lays out how to build an eval loop for an agentic Claude system: what to measure, how to score it when the right answer is fuzzy, and how to wire the whole thing into your release process so that quality is gated automatically rather than discovered in production.

Why agent evals are harder than test cases

A unit test checks a deterministic output against an expected value. Agent evals cannot work that way, for two reasons. First, agents are non-deterministic - the same input can produce different valid trajectories, so an exact-match assertion fails on correct behavior. Second, agent quality is multi-dimensional: a run can reach the right final answer through a wasteful, expensive path, or take a clean path to a wrong answer. You have to evaluate both the destination and the journey.

An agent eval is a repeatable measurement of how well an agent accomplishes a representative task, scored across the dimensions that matter - final outcome, the trajectory it took, cost, and latency. That definition forces you to be explicit about what good means for your use case before you can measure it, which is itself half the value. Teams that struggle with evals usually struggle because they never pinned down success criteria, not because scoring is hard.

Building your eval set: golden tasks and trajectory checks

The foundation is a curated set of representative tasks - your golden set. Each task has an input (the prompt and any starting state), a definition of success, and ideally a known-good trajectory or set of acceptable outcomes. Build this set from real usage: the tasks your users actually bring, the edge cases that have bitten you, and the failure modes you have already fixed and never want to see again. A good eval set is mostly cases that came from real pain.

The diagram shows how a single eval task flows from input to a pass or fail verdict.

flowchart TD
  A["Golden task: input + success criteria"] --> B["Run agent, capture full trajectory"]
  B --> C["Deterministic checks: did tool fire? final state correct?"]
  C --> D{"All hard checks pass?"}
  D -->|No| E["Fail: record trace for review"]
  D -->|Yes| F["LLM judge scores quality and reasoning"]
  F --> G{"Score above threshold?"}
  G -->|No| E
  G -->|Yes| H["Pass: record cost and latency"]

Score with the cheapest reliable method first. Wherever you can check something deterministically, do - did the agent call the refund tool, did the final database state match the expected state, did it stay under the step budget, did it avoid the forbidden action? These programmatic assertions are fast, free, and unambiguous, and they should carry as much of your eval as possible. Trajectory checks like the agent must verify the order before refunding it are some of the most valuable, because they catch dangerous shortcuts that a final-answer check would miss entirely.

Scoring fuzzy outputs with an LLM judge

Plenty of agent outputs cannot be checked with an equality assertion - a summary, a drafted email, an explanation. For these, the practical tool is an LLM judge: a separate Claude call whose job is to score a candidate output against a rubric. The judge receives the task, the agent's output, and explicit grading criteria, and returns a structured verdict - a score and a short justification - rather than a vague impression.

An LLM judge is a model prompted to evaluate another model's output against a defined rubric, returning a structured score. The quality of the judge depends almost entirely on the rubric: vague criteria produce noisy, unreliable scores, while specific, decomposed criteria - does it answer the actual question, is it factually grounded in the provided context, is the tone appropriate - produce scores you can trust. Validate your judge against human ratings on a sample before you rely on it; a judge that disagrees with your team is worse than no judge, because it gives false confidence.

Keep judges honest with a few habits. Ask for the reasoning before the score so the judgment is grounded. Use a capable model for judging even if the agent itself runs on a smaller one, since grading is often harder than the task. And remember that the judge is itself non-deterministic, so treat its scores statistically - aggregate across the eval set rather than agonizing over any single borderline case.

Gating releases: turning evals into a quality gate

An eval set that you run by hand once a month is a nice-to-have. An eval set that runs automatically on every change and blocks the bad ones is the actual product. The goal is a quality gate: before any prompt change, tool change, or model swap ships, the full eval suite runs, and the change is blocked unless it clears your thresholds.

Wire it into CI. When someone opens a change to the agent - a new system prompt, a reworded tool description, a model upgrade - the pipeline runs the golden set against the modified agent and reports pass rates, average quality scores, cost, and latency, alongside the current production baseline. A change that improves answer quality but doubles token cost is now a visible, explicit trade-off rather than a silent surprise. A change that regresses even one safety-critical trajectory check fails the gate outright, no matter how good the average looks.

Set thresholds deliberately. Some checks are hard gates - a safety trajectory that must never regress fails the build on a single violation. Others are statistical - an average quality score that must stay within a band, accepting that non-determinism means it will wobble. Track the numbers over time so you can see slow drift, the kind where no single change is bad but ten changes together quietly erode quality. The dashboard of pass rate, cost, and latency across releases is what turns agent development from anecdote into engineering.

Closing the loop: evals feed development

Evals are not a one-time gate; they are a flywheel. Every production failure becomes a new eval case, so the suite grows toward the exact shape of your real workload and the same bug can never ship twice. Every eval failure you investigate teaches you something about your prompts or tools that you fold back into the system. Over time the eval set becomes the most accurate specification of what your agent is supposed to do - more accurate than any document, because it is executable.

This is how the demo-to-production gap finally closes. Instead of shipping changes and hoping, you ship changes that have already cleared a representative gauntlet, and you watch production for new failure shapes to feed back in. The teams that ship reliable Claude agents are not the ones who write perfect prompts on the first try; they are the ones whose eval loop catches the imperfect ones before customers do.

Frequently asked questions

How many eval cases do I need to start?

Start small and real - even a dozen golden tasks drawn from actual usage and past failures will catch more regressions than zero. Grow the set every time production surprises you. Coverage of your real failure modes matters far more than raw count, so prioritize the cases that have actually hurt you.

Can I trust an LLM judge to score my agent?

You can, if you validate it. Write a specific, decomposed rubric, ask the judge for reasoning before its score, and check its verdicts against human ratings on a sample first. Use a capable model for judging and aggregate scores across the set rather than trusting any single borderline call, since the judge is itself non-deterministic.

How do I evaluate the path an agent took, not just the answer?

Capture the full trajectory and assert on it programmatically - did the required tool fire, did the agent verify before acting, did it stay under the step budget, did it avoid forbidden actions. These trajectory checks catch dangerous shortcuts that reach a correct-looking answer through an unsafe path, which a final-answer check would miss.

Should evals run in CI?

Yes. The value of evals comes from running them automatically on every change to the prompt, tools, or model, and blocking changes that regress your thresholds. Report pass rate, quality scores, cost, and latency against the production baseline so trade-offs are visible, and treat safety-critical checks as hard gates that fail the build on a single violation.

Evals behind every conversation

The same eval discipline - golden tasks, trajectory checks, LLM judges, and a CI quality gate - is what lets a voice agent change and improve without quietly breaking calls. CallSphere runs this kind of evaluation loop behind multi-agent voice and chat assistants that answer every call, use tools live, and book work around the clock. See how measured quality ships at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.