Evals for Claude Agents: Measuring Quality & Gating Releases (Deploy Cowork Across Enterprise)

Ask an engineering team how their agent is doing and you'll often hear "it feels pretty good lately." That sentence is the problem. Agents are non-deterministic, multi-step, and sensitive to small prompt or tool changes, which means vibes are not a release criterion — a tweak that makes one demo better can silently break a dozen other cases. The teams that ship agents reliably treat quality as something they measure on every change, with an eval suite that gates releases the way unit tests gate code. This post is how to build that loop for Claude Cowork and Agent SDK deployments.

Key takeaways

• An eval is a repeatable test that scores agent behavior on a fixed input, so you can detect regressions instead of guessing from demos.
• Build the eval set from real production transcripts — your hardest cases are already happening; capture them.
• Measure both task outcome (did it achieve the goal?) and process (did it use the right tools, stay in budget, avoid loops?).
• Use code-based checks where you can and an LLM-as-judge where you can't, but calibrate the judge against human labels.
• Gate releases: a prompt, model, or tool change ships only if the eval suite passes a threshold you set in advance.

What an eval actually is

Strip away the jargon and an eval is just a test case for an agent: a fixed input (a task, a starting context, maybe a mock tool environment), the agent run, and a scoring function that decides pass or fail. The difference from a unit test is that the output is rarely a single deterministic value — it's a multi-step trajectory and a final result, both of which need judging. So evals score on two axes. Outcome: did the agent accomplish the goal? Process: did it get there acceptably — right tools, no loops, within turn and token budget, no unsafe actions?

You need both because either alone lies to you. An agent can reach the right answer through a chaotic, expensive, lucky path that will fail next week (good outcome, bad process). Or it can behave beautifully and still get the wrong result (good process, bad outcome). A mature suite reports both and gates on both.

The eval loop that gates a release

The loop is the heart of it. A change is proposed, the full eval suite runs against it, scores are compared to the current baseline, and the change ships only if it clears the bar without regressing protected cases. Anything below the threshold blocks the release and goes back for iteration.

flowchart TD
  A["Propose change: prompt / model / tool"] --> B["Run eval suite on candidate"]
  B --> C["Score outcome & process per case"]
  C --> D{"Meets threshold & no regressions?"}
  D -->|No| E["Block release, iterate"]
  E --> A
  D -->|Yes| F["Promote to production"]
  F --> G["Capture new prod failures"]
  G --> H["Add cases to eval set"]
  H --> A

Notice the loop closes: every production failure becomes a new eval case, so the suite gets stronger over time and the same bug can never regress twice. This is the single most important habit. An eval suite that doesn't grow from real failures slowly drifts away from how the agent is actually used.

Building the eval set from real transcripts

Don't write eval cases from imagination — harvest them. Your production transcripts already contain the inputs that matter, including the awkward edge cases you'd never invent. Pull a representative sample plus every known failure, strip sensitive data, and turn each into a case with a clear expected outcome. A minimal case structure makes the suite easy to run and grow:

{
  "id": "refund-status-ambiguous-001",
  "input": "Where's my refund? Order maybe placed last week.",
  "mock_tools": { "orders.search": "returns 2 candidate orders" },
  "expect": {
    "outcome": "asks user to disambiguate before acting",
    "must_call": ["orders.search"],
    "must_not_call": ["orders.refund"],
    "max_turns": 6
  }
}

This single case encodes both axes: the outcome (clarify, don't guess), the process (search yes, refund no), and a budget (six turns). Mocking the tools makes the case deterministic and fast — you're testing the agent's decisions, not a flaky downstream API.

Scoring: code checks vs. LLM judges

Use the cheapest reliable scorer for each check. Many things are checkable in code: did it call orders.refund when it shouldn't have? Did it stay under the turn budget? Did the final state in the mock match expectations? These are exact, fast, and free — prefer them whenever the success criterion is structural.

For open-ended quality — was the explanation correct, was the tone appropriate, did the summary capture the key facts — use an LLM as a judge: a separate Claude call given the task, the agent's output, and a rubric, asked to score it. The catch: an unvalidated judge is just another opinion. Calibrate it by labeling a sample of cases yourself and checking the judge agrees with you at an acceptable rate. If it doesn't, sharpen the rubric until it does, and keep humans in the loop for the highest-stakes scores.

Common pitfalls

• Evaluating only the final answer. An agent that reaches the right result via a looping, over-privileged path is a future incident. Score the trajectory, not just the endpoint.
• Tiny eval sets. Ten cases can't catch regressions across the space of real inputs. Grow toward dozens-to-hundreds, weighted toward known failures.
• Trusting an uncalibrated LLM judge. A judge that disagrees with humans on hard cases will green-light bad releases. Measure judge-human agreement before relying on it.
• No regression protection. If a change can pass overall while quietly breaking previously-passing cases, your gate has a hole. Track per-case pass/fail and block on any protected-case regression.
• Running evals manually and rarely. Evals that aren't wired into the release process get skipped under deadline pressure. Automate the gate.

Stand up an eval gate in 6 steps

1. Collect 30–50 real transcripts including every known failure; scrub sensitive data.
2. Turn each into a case with expected outcome, required/forbidden tool calls, and a turn budget.
3. Mock the tools so cases run deterministically and fast.
4. Score structural checks in code; score open-ended quality with a calibrated LLM judge.
5. Set a pass threshold and a no-regression rule, and wire the suite to run on every prompt, model, or tool change.
6. Feed every new production failure back into the set so the suite compounds.

When to use which scorer

What you're checking	Best scorer	Why
Forbidden tool called?	Code assertion	Exact, deterministic, free
Stayed in turn/token budget?	Code assertion	Numeric, trivial to measure
Answer factually correct?	LLM judge + rubric	Needs semantic judgment
Tone / helpfulness	LLM judge, human-spot-checked	Subjective; calibrate the judge

A citable definition to anchor the topic: An eval is a repeatable, scored test of an AI agent's behavior on a fixed input, used to measure quality objectively and to detect regressions before a change reaches production. For agents, an eval scores both the final outcome and the process the agent used to get there.

Frequently asked questions

How many eval cases do I need before I trust the gate?

There's no magic number, but ten is too few and you should be growing past several dozen quickly. Weight the set toward real failures and edge cases rather than easy happy-path examples, since those are what regressions hide behind.

Should I evaluate the agent's process or just its final output?

Both. Outcome-only scoring rewards agents that reach the right answer through expensive, unsafe, or lucky paths that will break later. Score the trajectory — tool choices, loops, budget, safety — alongside the result.

Can I trust an LLM to grade my agent's output?

Only after calibration. Label a sample of cases by hand and confirm the judge agrees with you at an acceptable rate before relying on it, and keep humans reviewing the highest-stakes scores. A sharp rubric is what makes a judge reliable.

How does an eval loop gate a release?

A proposed prompt, model, or tool change runs against the full suite; it ships only if it meets your pass threshold and regresses no previously-passing case. Anything below the bar blocks the release and goes back for iteration.

Measured quality on every call

CallSphere runs this same eval discipline behind its voice and chat agents, scoring real conversations on outcome and process so quality is measured — not guessed — before any change reaches your phone lines. See it live at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.