Evals for Claude Agents: Measuring Quality & Gating Ships

Most agent teams ship on vibes. Someone tries a few prompts, the agent seems smarter than last week, and the change goes out. Then a customer hits an edge case that the "smarter" prompt quietly broke, and there's no way to know whether the regression was introduced last week or six months ago. The cure is the same one that saved software engineering decades ago: a test suite. For agents, that test suite is an eval loop, and building one is the difference between guessing and knowing.

This post is about constructing evals for agents built on the Claude Agent SDK — what to measure, how to score it when the right answer is fuzzy, and how to wire the result into a release gate so a quality regression can't reach production unnoticed. Evals are unglamorous infrastructure, and they're the single biggest predictor of whether an agent stays reliable as it changes.

Why agent evals are harder than unit tests

A unit test asserts that add(2, 2) equals 4. An agent eval has to assert something far slipperier: that given a customer asking to reschedule an appointment, the agent gathered the right information, called the right tools in a sensible order, and produced a correct, appropriately-toned response. There's rarely one exact right string, the agent can reach a good outcome by different valid paths, and quality is partly a judgment call. That's why naive string-matching fails almost immediately for agents.

The definition to anchor on: an agent eval is a repeatable test that runs the agent against a fixed input and scores its behavior against defined quality criteria — checking not just the final answer but the trajectory, meaning which tools were called, with what arguments, and in what order. Trajectory matters because two agents can give the same final answer while one took a safe, cheap path and the other made six wasteful or risky tool calls to get there.

What to measure: outcome and trajectory

Split your metrics into outcome quality and trajectory quality. Outcome quality asks: was the final answer correct, complete, and appropriately worded? Trajectory quality asks: did the agent call the right tools, avoid forbidden ones, pass valid arguments, and finish in a reasonable number of steps? A good eval suite checks both, because an agent that gets the right answer the wrong way is a future incident.

flowchart TD
  A["Eval dataset: inputs + criteria"] --> B["Run agent on each case"]
  B --> C["Capture final answer + tool trajectory"]
  C --> D["Deterministic checks: tools used, args valid"]
  C --> E["LLM judge scores answer quality"]
  D --> F["Aggregate score per case"]
  E --> F
  F --> G{"Score >= release threshold?"}
  G -->|No| H["Block release, surface failing cases"]
  G -->|Yes| I["Promote build"]

The diagram shows the two scoring lanes feeding one decision. Deterministic checks handle everything you can assert in code — was a required tool called, were arguments well-formed, did the run stay under the step cap, did a forbidden tool stay unused. These are cheap, fast, and never flaky. Reserve the slower, fuzzier judgment for the things code genuinely can't check.

Using an LLM as a judge — carefully

For the subjective half — is this answer helpful, accurate, and correctly toned? — use a capable model like Opus 4.8 as a judge, scoring each response against an explicit rubric. The key to making a judge trustworthy is the rubric: don't ask "is this good?", ask specific, scorable questions — "Does the response correctly answer the customer's actual question? Does it avoid claiming facts not present in the tool results? Is the tone professional?" — each with a defined scale.

Judges have failure modes you must control. They can be inconsistent run to run, so pin the rubric and keep the judge prompt stable. They can drift if you change the judge model, so version it alongside your evals. And you should calibrate the judge against human labels periodically — have a person score a sample of the same cases and confirm the judge agrees, so you're not gating releases on a scorer nobody validated. A judge is a tool, not an oracle.

Building the dataset that makes evals real

An eval suite is only as good as its cases. Start by mining production: every bug you fix, every weird transcript, every customer complaint becomes a permanent eval case so it can never silently regress. Add the hard edge cases you know the agent struggles with — ambiguous requests, missing information, adversarial inputs — because a suite of only easy cases will pass while real users suffer. Aim for coverage across the behaviors that matter, not just volume.

Keep cases versioned and reviewed like code, and label them by category so a failure tells you what broke — routing, tone, tool arguments, refusal handling. When you change a prompt or swap a model, run the full suite and read the diff: which categories improved, which regressed. This turns a scary, opaque change ("new system prompt") into a measured one ("refund-routing accuracy up 8 points, tone unchanged, no regressions").

Gating releases on the score

The final piece is the gate. Wire your eval suite into CI so every change to the prompt, tools, or model triggers a full run, and set a threshold below which the build cannot ship. The threshold should be concrete — an aggregate score, plus hard rules like "zero failures in the safety category" and "no regression greater than a few points in any category." A failing eval should block the merge the same way a failing unit test does.

Resist the urge to ratchet the threshold down when a change you like fails the gate. The gate only protects you if it's binding. When a legitimately better build trips an old case because the expected behavior genuinely changed, update the case deliberately and in review — don't lower the bar. Over time this loop compounds: each release makes the suite a little more comprehensive, and the agent gets measurably, not anecdotally, better.

Frequently asked questions

Why can't I just check the agent's final answer?

Because two agents can produce the same answer via very different paths — one safe and cheap, one wasteful or risky. Trajectory checks (which tools were called, with what arguments, in what order) catch the agent that got lucky on the answer while doing something you'd never want repeated at scale.

Is using an LLM to grade an agent reliable?

It's reliable enough to gate on if you control it: use an explicit scorable rubric, pin the judge model and prompt, and periodically calibrate against human labels. Use deterministic code checks for anything objectively verifiable and reserve the judge for genuinely subjective quality dimensions.

How many eval cases do I need to start?

Fewer than you think — a few dozen well-chosen cases covering your riskiest behaviors beats hundreds of trivial ones. Grow the suite by converting every production bug and edge case into a permanent case, so coverage tracks the failures that actually happen rather than the ones you imagined.

What should block a release?

Any drop below your aggregate threshold, any failure in a safety or compliance category, and any meaningful regression in a tracked category. Treat the gate as binding — fix the agent rather than lowering the bar, and only update expected behavior in a case when the change is deliberate and reviewed.

Bringing agentic AI to your phone lines

CallSphere holds its voice and chat agents to the same bar — trajectory checks, rubric-scored quality, and release gates — so the assistants that answer every call and book work 24/7 stay reliable as they evolve. See it live at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.