Evals for Claude Agents: Gate Every Release

Here's a question that separates teams who ship reliable agents from teams who don't: when you change your system prompt, how do you know you didn't make things worse? If the answer is "we tried a few examples and it looked fine," you don't have a quality process — you have a vibe. Agents are non-deterministic, their behavior shifts in surprising ways when you tweak a tool description or swap a model, and a change that fixes one case routinely breaks three you weren't looking at. Without evals, you are flying blind, and you will ship regressions to users.

An eval loop fixes this. It turns "does this agent work?" from an opinion into a measurement. You build a dataset of representative tasks with known-good outcomes, you run the agent against them, you score the results automatically, and you make that score a gate: a change can't ship unless it holds or improves the number. This is the same idea as a test suite, adapted to the reality that the thing under test is probabilistic. This post is a practical guide to building that loop for Claude agents.

Key takeaways

• An eval is a dataset of tasks plus an automated grader that scores the agent's output, run on every change to catch regressions.
• Build your dataset from real failures and real traffic, not invented happy-path cases.
• Choose graders by task: exact/programmatic checks when there's a right answer, LLM-as-judge for open-ended quality.
• Grade the trajectory, not just the final answer — did the agent call the right tools in a sane order?
• Wire evals into CI as a release gate with a clear pass threshold so regressions can't merge.

What to measure: outcomes and trajectories

Agents have two things worth grading, and beginners measure only the first. The obvious one is the final outcome: did the agent produce the right answer, book the correct slot, return the right data? The subtler one is the trajectory: the sequence of tool calls it made to get there. An agent can stumble into the right answer through a wasteful, lucky path that will fail on the next input, so trajectory matters.

Concretely, a good eval for a booking agent checks both that the meeting landed on the right calendar at the right time (outcome) and that the agent looked up availability before booking rather than guessing (trajectory). Outcome checks catch wrong results; trajectory checks catch fragile reasoning that hasn't failed yet. Mature eval suites assert on both, because the trajectory is your early warning system.

flowchart TD
  A["Code or prompt change"] --> B["CI runs eval suite"]
  B --> C["Agent runs each dataset case"]
  C --> D["Grade outcome & trajectory"]
  D --> E{"Score >= threshold?"}
  E -->|Yes| F["Merge / deploy"]
  E -->|No| G["Block + show failing cases"]
  G --> H["Fix, add case, rerun"]
  H --> B

Building a dataset that actually represents reality

The quality of your eval is the quality of your dataset, and the most common failure is filling it with easy, invented cases the agent was always going to pass. Those tell you nothing. The cases that move quality are the hard ones: real inputs from production, edge cases that broke the agent before, ambiguous requests, adversarial phrasings, and the long tail of weird-but-real user behavior.

The best source of cases is your own failure log. Every time the agent gets something wrong in production, capture the input, decide what the correct behavior was, and add it to the dataset as a regression case. Over time this gives you an eval that's relentlessly focused on your actual weak spots. Aim for breadth across categories — happy path, edge cases, ambiguity, and known past failures — and keep each case small enough to debug when it fails.

Choosing graders: programmatic vs. LLM-as-judge

How you score depends on whether there's a single right answer. When the output is structured or verifiable — an extracted value, a chosen tool, a booked time, a number — use a programmatic grader: a plain assertion in code. These are fast, free, deterministic, and impossible to argue with. Always prefer them when the task allows.

When the output is open-ended — a summary, an explanation, a customer reply — there's no string to match against, so you use LLM-as-judge: a second Claude call that scores the output against a rubric you write. The rubric is everything. A vague "rate this 1–10" produces noisy, useless scores; a specific rubric that lists exactly what a good answer must contain produces consistent ones. Here's a usable judge rubric:

You are grading a support agent's reply. Score PASS or FAIL.
PASS requires ALL of:
  1. Directly answers the customer's actual question.
  2. Contains no invented facts (only info from the provided context).
  3. Offers a concrete next step.
  4. Tone is professional and warm.
Return JSON: { "verdict": "PASS" | "FAIL", "reason": "" }
Reply to grade:
---
{agent_output}

Binary PASS/FAIL with a required-criteria list is far more reliable than a numeric scale, because it forces the judge to make a defensible call. Validate your judge by spot-checking its verdicts against human judgment on a sample — a miscalibrated judge gives you false confidence.

Wiring evals into a release gate

An eval suite that runs only when someone remembers to run it is worthless. The point is to make it automatic and blocking. Wire the suite into CI so that every change — prompt edit, tool change, model swap — triggers a run, and set a pass threshold the change must meet to merge. When the score drops, CI fails and shows the failing cases, turning a silent regression into a loud, fixable one.

Two practical notes. First, account for non-determinism: run flaky-prone cases a few times and require a majority pass, or set thresholds with a little margin so normal variance doesn't block good changes. Second, track the score over time. A slow downward drift across many small changes is a signal you'd miss looking at any single PR. Treat your eval score like a key SLO for the agent.

Common pitfalls

• Happy-path-only datasets. Cases the agent always passes measure nothing. Mine real failures and edge cases.
• Grading only the final answer. A right answer via a broken path will fail next week. Assert on the trajectory too.
• Vague LLM-judge rubrics. "Rate 1–10" is noise. Use binary PASS/FAIL with explicit required criteria.
• An unvalidated judge. If you never check the judge against humans, you may be gating on garbage. Spot-check it.
• Evals that don't block. If a failing eval can still merge, it isn't a gate — it's decoration.

Stand up an eval loop in five steps

1. Collect 20–50 representative cases — real inputs, edge cases, and every past production failure.
2. For each case, define the correct outcome and the expected tool trajectory.
3. Pick graders: programmatic assertions where there's a right answer, an LLM judge with a strict rubric where it's open-ended.
4. Run the suite in CI on every change and set a blocking pass threshold.
5. Every new production failure becomes a new eval case — grow the suite forever.

Task type	Grader	Why
Extraction / classification	Programmatic assertion	One verifiable right answer
Tool selection	Trajectory check	Assert the right tool was called
Summary / reply	LLM-as-judge (rubric)	Open-ended, no exact match
End-to-end task	Outcome + trajectory	Result and path both matter

Frequently asked questions

What is an eval for an AI agent?

An eval is a dataset of representative tasks paired with an automated grader that scores the agent's output and trajectory. Run on every change, it turns "does this work?" into a measurement and catches regressions before they reach users.

When should I use LLM-as-judge versus a code assertion?

Use a programmatic assertion whenever the task has a verifiable right answer — an extracted value, a chosen tool, a booked time. Use an LLM judge with a strict rubric only for open-ended outputs like summaries or replies where no exact match exists.

How many eval cases do I need?

Start with 20–50 well-chosen cases covering happy path, edge cases, ambiguity, and past failures — quality and coverage beat raw count. Then grow the suite continuously by adding every new production failure as a case.

How do I handle non-determinism in evals?

Run variance-prone cases several times and require a majority pass, set thresholds with a small margin, and pin the model version. Track the aggregate score over time so slow drift across many changes becomes visible.

Bringing agentic AI to your phone lines

CallSphere holds its voice and chat agents to the same bar — every prompt and tool change runs through an eval gate before it touches a live call, so quality goes up and never quietly slips. See it live at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.