Evals for Claude Cowork: Gating a Sales-Book Release

Here's the uncomfortable truth about running an agent over four thousand accounts: you cannot eyeball the output. Spot-checking a dozen records tells you nothing about the other 3,988, and the failures that hurt — a wrong field on one percent of accounts — are exactly the ones a casual review misses. The only way to ship changes to a Claude Cowork workflow with confidence is to measure quality the way you'd measure any other system: with an eval loop that scores runs, sets a bar, and refuses to ship anything below it.

Why agents need evals more than prompts need vibes

Every change you make to an agentic workflow — a tweaked instruction, a new tool, a model upgrade — can shift behavior in ways that are invisible until they cost you. A prompt edit that fixes one edge case can silently break three others. Without evals, you're tuning a system you can't see, and "it looked fine on the examples I tried" is not a quality bar. An eval is a repeatable, scored test of agent behavior against a known-good standard — the agentic equivalent of a regression test suite, and just as non-negotiable for anything running at scale.

The hardest part is that agent quality is multi-dimensional. A run can be correct (right data written), safe (no forbidden actions), efficient (reasonable token cost), and complete (no accounts skipped) — or it can ace some dimensions while quietly failing others. Your eval has to measure all of them, because a workflow that's accurate but burns ten times the budget, or cheap but corrupts one account in fifty, is not actually shippable.

Building the golden set

Evals start with a golden set: a representative sample of accounts with known-correct outcomes. Don't pick easy records — pick the ones that exercise the hard paths. Include accounts with missing fields, duplicate contacts, unusual histories, and the stage transitions where mistakes are most likely. A hundred well-chosen accounts that cover your real distribution beat a thousand random ones. For each, write down what "correct" looks like: which fields should change, to what values, and which actions should and shouldn't fire.

The golden set is an asset you grow over time. Every time the agent fails in production on a record you didn't anticipate, you fix it and then add that record to the golden set. The eval suite becomes a living memory of every mistake the system has ever made, and a guarantee that it won't make any of them again. This is the flywheel that separates a workflow that gets more reliable over time from one that breaks in a new way every week.

flowchart TD
  A["Proposed change to workflow"] --> B["Run agent over golden set"]
  B --> C["Score: correctness, safety, cost, completeness"]
  C --> D{"All metrics >= threshold?"}
  D -->|No| E["Block release, inspect failures"]
  E --> A
  D -->|Yes| F["Promote to canary: 200 live accounts"]
  F --> G{"Canary healthy?"}
  G -->|No| E
  G -->|Yes| H["Roll out to full book"]

Scoring: deterministic checks plus a judge

How do you score a run? Two complementary methods. The first is deterministic checks — code that compares the agent's actual writes against the golden answers. Did the right field get the right value? Was a forbidden tool called? Were any accounts skipped? These are fast, free, and unambiguous, and they should cover everything that has an objectively correct answer. Most of your eval signal should come from here.

The second is an LLM judge for the things deterministic checks can't capture — was the re-engagement note the agent drafted actually appropriate and on-brand? You give a model the account context, the agent's output, and a rubric, and ask it to score against that rubric. Judges are powerful but fallible, so calibrate them: have a human grade a sample, compare to the judge's scores, and refine the rubric until they agree. Use the judge only where deterministic checks fall short, and treat its score as one input rather than gospel.

Gating releases with a quality bar

Evals only matter if they can say no. The point of the loop is a gate: a proposed change runs against the golden set, gets scored on every dimension, and is blocked from release unless it clears the threshold on all of them. Set the bar deliberately — for a sales book that might mean ninety-nine percent field-level correctness, zero forbidden actions, and cost per account within a defined ceiling. A change that improves note quality but pushes cost over the ceiling doesn't ship until that's resolved.

Wire this into your process so it's automatic, not a thing someone remembers to do. Before any workflow change reaches production, the eval runs and either passes or blocks. This catches the insidious failures — the model upgrade that subtly changes formatting, the new tool that the agent over-uses — before they touch a single real customer. The gate is what lets you move fast: you can change the workflow freely because the eval will catch you if you break something.

Canary releases and production monitoring

Even a perfect eval can't anticipate everything about live data, so don't go from golden set to full book in one jump. Use a canary: run the new version against a small slice of real accounts — a couple hundred — and watch the same quality and cost metrics you measure offline. If the canary stays healthy, roll out to the full book; if it doesn't, you've contained the damage to two hundred accounts instead of four thousand.

Finally, monitoring is the eval that never stops. Track correctness signals in production — review-queue rate, write-rejection rate, cost per account — and alert when they drift. Production is where you discover the cases your golden set missed, and each one becomes a new test. Offline evals gate releases; production monitoring feeds the golden set; the two together form a loop that makes the system measurably more reliable every week instead of quietly degrading.

Frequently asked questions

What exactly is an eval for an agentic workflow?

It's a repeatable, scored test of agent behavior against a known-good standard — the agentic equivalent of a regression suite. You run the agent over a curated set of accounts with known-correct outcomes and score the result on correctness, safety, cost, and completeness, then use that score to decide whether a change can ship.

How big should my golden set be?

Coverage matters more than size. A hundred accounts that deliberately exercise the hard paths — missing fields, duplicates, tricky stage transitions — beat a thousand random easy ones. Grow the set over time by adding every production failure you encounter, so the suite becomes a memory of every mistake the system has made.

Should I use an LLM as a judge to score runs?

Use deterministic code checks for anything with an objectively correct answer — they're fast, free, and unambiguous — and reserve an LLM judge for subjective qualities like whether a drafted note is appropriate. Calibrate the judge against human grades before trusting it, and treat its score as one input among several.

How does an eval gate a release?

The proposed change runs against the golden set and must clear a threshold on every metric before it ships. If correctness, safety, or cost falls below the bar, the release is blocked. Pair the offline gate with a canary run on a few hundred live accounts to catch anything the golden set didn't cover before full rollout.

Bringing agentic AI to your phone lines

Golden sets, scored runs, and release gates are exactly how you keep a customer-facing agent reliable as it evolves. CallSphere applies these agentic-AI evaluation patterns to voice and chat — assistants that answer every call and message, use tools mid-conversation, and book work 24/7 against a measured quality bar. See it live at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.