Evals for Claude Agents: Measuring Quality & Gating Releases

The teams at the Built-with-Opus hackathon split cleanly into two groups by Sunday: those who could say whether their latest change made the agent better, and those who could only say it "felt" better. The first group had evals. The second group was flying on vibes, and every prompt tweak was a coin flip — fix one case, silently break two others, and never know. If you take one thing from a weekend of building agents, take this: an agent without an eval loop is not engineering, it is gambling.

Testing agents is genuinely harder than testing functions. There is no single correct output, the same input can produce different valid trajectories, and quality is often a judgment call. But "harder" does not mean "skip it." It means you need a different toolkit — eval sets, rubric scoring, judge models, and a gate that blocks releases when quality regresses. This post is how the hackathon teams built that loop in two days.

Why you cannot test an agent like a function

A unit test asserts that a function returns an exact value. An agent might solve the same task three different ways, all acceptable, with different tool calls and different wording each time. Assert on the exact transcript and your test breaks on every harmless variation. Assert on nothing and you catch no regressions. The resolution is to test at the level of outcomes and behaviors, not exact strings.

For the refactor agent, the eval did not check the exact diff; it checked that the code still compiled, the tests still passed, and no forbidden file was touched. For the data agent, it checked that the answer contained the right figure within tolerance and cited a real source. These are assertions on what good looks like, robust to the many valid paths a capable model takes. That shift — from matching output to verifying properties — is the foundation of agent testing.

Building an eval set that actually represents your work

An eval set is a curated collection of representative tasks paired with a way to judge whether the agent handled each one well. The fastest way to build one is to mine real runs. Every time the agent failed during development, the teams captured that case — the input, what went wrong, and what right would have looked like — and added it to the set. Within a day, the eval set was a museum of every mistake the agent had ever made, which is exactly what you want it to never make again.

flowchart TD
  A["Code or prompt change"] --> B["Run agent over eval set"]
  B --> C{"Deterministic checks pass?"}
  C -->|No| D["Fail fast: report broken case"]
  C -->|Yes| E["LLM judge scores against rubric"]
  E --> F{"Score >= release threshold?"}
  F -->|No| G["Block release, surface regressions"]
  F -->|Yes| H["Promote build"]

The diagram is the gate the teams converged on. A change triggers a full run over the eval set. Cheap deterministic checks run first — does it compile, do tests pass, were forbidden actions avoided — and fail fast because they are free. Only the cases that survive go to the more expensive LLM judge. The judge's aggregate score is compared against a release threshold, and anything below it blocks the build. The ordering matters: never spend judge tokens on a case that already failed a deterministic check.

Scoring: deterministic checks plus LLM judges

Two kinds of checks cover most needs. Deterministic checks are programmatic assertions — exit codes, regex matches, schema validation, "did the test suite pass." They are cheap, fast, and unambiguous, so use them for everything you can express as a rule. The data agent's "is the number correct within tolerance" was a deterministic check, and it caught most regressions for free.

For the fuzzy parts — was the explanation clear, was the tone right, did the summary capture the key point — you need a judge. The pattern is to use Claude as an LLM judge: give a strong model the task, the agent's output, and a rubric, and ask it to score each criterion with a short justification. The justification matters as much as the score, because it tells you why a case regressed. A good rubric is specific: not "is this good" but "does the response cite a source, stay under 200 words, and avoid speculation." Vague rubrics produce noisy judges; precise rubrics produce stable, trustworthy scores.

Gating releases on the eval loop

An eval set you run occasionally is a nice-to-have. An eval set wired into your release process is a safety net. The teams that shipped confidently made the gate non-optional: no change to the agent's prompt, tools, or model shipped until it ran the full eval set and cleared the threshold. That turned every change from a gamble into a measured decision — the dashboard either showed the score went up, held, or dropped, and you acted accordingly.

Set the threshold honestly. A brittle 100%-pass gate trains people to disable the gate; a threshold that reflects real acceptable quality, plus a hard rule that scores must never drop on previously-passing cases, keeps the bar meaningful. The most useful signal was not the absolute number but the delta: this change moved the score from 82 to 79, here are the three cases that regressed, do we accept the trade. That conversation is impossible without the loop, and trivial with it.

Avoiding evals that lie to you

Evals can give false confidence, and the hackathon surfaced the common traps. The first is an eval set that is too small or too easy — it passes everything and catches nothing. Grow it from real failures and keep adding hard cases. The second is overfitting: if you tune the agent until it aces the eval set, you may have memorized the test rather than improved the agent. Hold out a portion of cases you never tune against, and check the agent against those before shipping.

The third trap is a careless judge. An LLM judge with a vague rubric, or one biased toward longer or more confident answers, will reward the wrong things. Calibrate the judge by hand-scoring a sample and checking that it agrees with you; if it does not, the rubric needs work, not the agent. Treat the judge as a component to be tested, not an oracle to be trusted.

Make the loop fast enough to actually use

The final lesson: an eval loop only helps if it runs often, and it only runs often if it is fast and cheap. The teams kept a small, sharp "smoke" eval set that ran in seconds on every change, and a larger comprehensive set that ran before release. Deterministic checks first, judges only where needed, and caching on the static parts of judge prompts kept costs low. With a loop that returns a verdict in under a minute, engineers actually used it on every change — which is the entire point. Quality you measure constantly is quality you can improve deliberately.

Frequently asked questions

What is an eval set for an AI agent?

An eval set is a curated collection of representative tasks paired with a way to judge whether the agent handled each one well. The best ones are built from real failures captured during development, and they combine deterministic checks for rule-based criteria with LLM-judge scoring for fuzzy qualities like clarity and tone.

How does an LLM judge work for scoring agents?

You give a strong model — such as Claude — the original task, the agent's output, and a specific rubric, and ask it to score each criterion with a short justification. The justification reveals why a case passed or regressed. Precise rubrics produce stable scores; vague ones produce noisy, untrustworthy judgments, so calibrate the judge against hand-scored samples.

Why not just test agents like normal functions?

Because an agent can solve the same task several valid ways with different trajectories and wording. Asserting on exact output breaks on harmless variation. Instead, test at the level of outcomes and properties — did the code still compile, was the figure correct within tolerance, were forbidden actions avoided — which is robust to the many valid paths a capable model takes.

How do I gate a release on evals?

Make the eval run non-optional in your release process: no change to prompt, tools, or model ships until it clears the full eval set against a threshold, with a hard rule that previously-passing cases must not regress. Run deterministic checks first and fail fast, then send survivors to the judge, and act on the score delta.

Bringing agentic AI to your phone lines

A voice agent that talks to customers needs the same eval discipline before every change goes live. CallSphere gates its voice and chat agents behind measured quality loops so each release answers calls and books work better than the last, not worse. See it live at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.