Testing Claude Agents: Build an Eval Loop That Gates

Here is the uncomfortable truth about shipping agents: you changed one line of the system prompt to fix a bug, it fixed the bug, and it silently broke three other behaviors you did not think to check. Without an evaluation loop you have no way to know. Agents are non-deterministic, multi-step, and sensitive to tiny wording changes, which makes the test-and-pray workflow that works for ordinary code actively dangerous here. The team that ships agents confidently is the team that built evals first.

An eval is just a repeatable measurement of agent quality against examples you care about, run automatically so that a regression blocks a release before it reaches users. This post covers how to build that loop for Claude Managed Agents: what to measure, how to score it when there is no single right answer, and how to wire evals into a gate that protects production.

Key takeaways

• Evals are not optional for agents — non-determinism means you cannot eyeball quality or trust a single passing run.
• Score outcomes and behaviors, not exact text: did it reach the goal, call the right tools, and avoid forbidden actions.
• Use the right scorer per case: exact match for structured outputs, programmatic checks for tool use, and an LLM judge for open-ended quality.
• Run each case multiple times and track a pass rate; a flaky agent that passes once is not passing.
• Gate releases on the eval suite so a prompt or tool change cannot ship a regression.

Why a single run tells you nothing

Because an agent makes a chain of stochastic decisions, the same input can succeed brilliantly and fail embarrassingly across two consecutive runs. A demo that worked once proves only that success is possible, not that it is reliable. This is the core reason agent testing differs from ordinary testing: your unit of measurement is not a boolean but a rate. You ask "how often does this scenario succeed" and you only trust a change when the rate holds or improves across many runs.

That reframing drives everything else. You need a set of representative scenarios, a way to score each run that does not depend on exact wording, and enough repetitions to estimate a stable pass rate. The investment feels heavy at first, then pays for itself the first time the suite catches a regression you would otherwise have shipped.

What to measure: outcomes and behaviors

Agents rarely produce one canonical correct string, so asserting on exact output is a trap. Instead, measure two things. Outcomes: did the agent achieve the goal — was the ticket actually created, the right answer returned, the booking made? Behaviors: did it get there acceptably — did it call the correct tools, avoid forbidden actions, stay within a turn budget, and refuse when it should have refused?

flowchart TD
  A["Eval case: input + expectations"] --> B["Run agent N times"]
  B --> C["Collect transcripts + final outputs"]
  C --> D{"What kind of check?"}
  D -->|Structured output| E["Exact / schema match"]
  D -->|Tool behavior| F["Programmatic assertion on calls"]
  D -->|Open-ended quality| G["LLM judge with rubric"]
  E --> H["Aggregate pass rate"]
  F --> H
  G --> H
  H --> I{"Rate >= threshold?"}
  I -->|Yes| J["Allow release"]
  I -->|No| K["Block + report regressions"]

A single eval case bundles an input, the expected outcome, and the behavioral constraints. For example: input is a customer asking to reschedule; the expected outcome is that update_booking was called with the new time; the constraints are that no booking was deleted and the agent confirmed the change in its reply. That case now tests something real and survives rewording of the agent's prose.

Choosing a scorer per case

Different cases need different scorers, and mixing them up is the most common eval mistake. Use exact or schema matching when the agent must emit structured data — a JSON object, a classification label, a specific value. Use programmatic assertions for tool behavior: parse the transcript and check that the right tool was called with arguments in the expected shape, and that forbidden tools were never invoked. These checks are deterministic, cheap, and trustworthy — lean on them wherever the expectation can be expressed in code.

For genuinely open-ended quality — was the explanation clear, was the tone right, did the summary capture the key points — use an LLM as a judge. Give the judge a concrete rubric and ask it to score against specific criteria rather than a vague "is this good." The judge model reads the agent's output and the rubric and returns a structured verdict you can aggregate. A well-specified rubric is the difference between a useful judge and a coin flip.

{
  "input": "Reschedule my Tuesday 2pm appointment to Thursday 3pm",
  "expect_tool_called": "update_booking",
  "expect_args_contains": { "new_time": "Thursday 15:00" },
  "forbidden_tools": ["delete_booking", "cancel_account"],
  "judge_rubric": "Reply must confirm the new time and not invent details",
  "runs": 5,
  "pass_threshold": 0.8
}

This case runs five times and passes only if at least four runs call update_booking correctly, never touch a forbidden tool, and clear the judge's rubric. That is a far stronger signal than a single happy-path demo.

Building the dataset that actually catches bugs

Your eval suite is only as good as the cases in it, and the most valuable cases come from production. Every time the agent fails in the real world, capture that exact scenario, add it to the suite with the correct expected behavior, and you have permanently inoculated against that regression. Over time the suite becomes a memory of every mistake the agent has ever made — a far better dataset than anything you could invent up front.

Balance the suite across three categories: happy paths that should always work, known hard cases that previously failed, and adversarial cases that probe for unsafe behavior. Do not over-index on the happy path; a suite that only tests the easy scenarios passes right up until users find the hard ones. Aim for breadth of failure modes over volume of similar cases.

Gating releases on the eval loop

An eval suite that runs manually is a suggestion; an eval suite wired into your release pipeline is a gate. The pattern is straightforward: any change to the prompt, the tools, the model, or the orchestration triggers the full suite, and the release is blocked unless the aggregate pass rate meets your threshold and no critical case regressed. This turns "I think this prompt change is safe" into "the suite confirms it is safe."

Set thresholds per case importance. A core revenue path might require a perfect pass rate, while a nice-to-have feature tolerates the occasional miss. Track the trend over time, not just the snapshot — a slow erosion of pass rate across releases is a signal worth catching before it becomes a customer-visible problem. The gate is what lets a team iterate fast on agents without iterating straight into a regression.

Case type	Best scorer	Example assertion
Structured output	Exact / schema match	Output equals expected JSON
Tool usage	Programmatic check	update_booking called; delete never called
Open-ended prose	LLM judge + rubric	Reply confirms time, invents nothing
Safety / refusal	Programmatic + judge	Refuses out-of-policy request

Common pitfalls

• Asserting on exact agent prose. Wording varies run to run; assert on outcomes and tool behavior instead, and reserve text checks for structured fields.
• Running each case once. A single pass hides flakiness. Run multiple times and require a pass rate.
• A vague LLM judge. "Is this good?" produces noise. Give the judge a specific rubric and have it score named criteria.
• A happy-path-only suite. It passes until real users hit the hard cases. Pull failures from production into the suite continuously.
• Evals that never gate anything. If a failing suite cannot block a release, it is documentation, not protection.

Stand up an eval loop in 6 steps

1. Collect a starter dataset of real inputs, including any production failures you can recall, with expected outcomes.
2. For each case, define the expected outcome, required and forbidden tool calls, and any prose rubric.
3. Assign the right scorer per case: exact match, programmatic tool check, or LLM judge with a rubric.
4. Run every case multiple times and aggregate into a per-case pass rate with a threshold.
5. Wire the suite into your release pipeline so prompt, tool, or model changes trigger it and a regression blocks the ship.
6. Feed every new production failure back into the suite so the agent can never regress on it twice.

Frequently asked questions

How many runs per eval case is enough?

Enough to estimate a stable pass rate — a handful for routine cases, more for high-stakes ones where you need tighter confidence. The principle is that one run measures nothing; you want enough repetitions that the rate is not dominated by luck. Increase the count for critical paths and decrease it for cheap, low-risk checks.

Is an LLM judge reliable enough to gate releases?

It is reliable when constrained. Give it a specific rubric, ask it to score named criteria rather than make a holistic call, and validate the judge itself against human-labeled examples periodically. For pure correctness, prefer programmatic checks; reserve the judge for the open-ended quality dimensions code cannot capture.

What do I do when an eval is flaky — sometimes passing, sometimes failing?

Flakiness is information: the agent's behavior on that case is genuinely unreliable, which is a quality bug, not a test bug. Investigate the failing transcripts, find why the agent diverges, and fix the underlying cause — usually unclear context or an ambiguous tool. A consistently flaky case should lower your confidence in shipping.

Can I reuse the same suite across model upgrades?

Yes, and you should — the suite is exactly how you decide whether a new model version is safe to adopt. Run the existing eval set against the new model and compare pass rates case by case. That turns a model migration from a leap of faith into a measured comparison with clear regressions surfaced.

Bringing agentic AI to your phone lines

CallSphere gates its voice and chat agents behind the same eval discipline — outcome checks, tool assertions, and rubric-scored quality — so every release that answers calls and books work has been measured before it ships. See it live at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.