Evals for Claude Opus agents: gate releases with an eval loop

You can't ship an agent you can't measure. A coding agent that passes your manual spot-check on Tuesday can regress silently on Thursday when you tweak the system prompt, bump the model, or add a tool — and without an eval suite, you'll learn about it from a user, not from CI. Evals are how you turn "it seems better" into a number you can defend, and an eval loop is how you stop a regression before it reaches production. With Claude Opus 4.8 as the agent and a second Claude model as the judge, building that loop is straightforward; the hard part is designing evals that actually correlate with quality. This post covers both.

What an eval actually is

An eval is a fixed input, an automated way to score the output, and a threshold the score must clear. For agents, the input is a task — "refactor this function to remove the duplicate validation" — and the output is whatever the agent produced: the final answer, the files it changed, the tool calls it made along the way. The scoring is where teams stall, because agent outputs are open-ended. There's no single correct string to diff against. The way through is to grade against explicit, checkable criteria rather than a golden answer: does the refactored code still pass the existing tests, did it touch only the intended function, did it avoid adding unrequested abstractions.

Those criteria become a rubric, and the rubric is the heart of a good eval. Vague criteria — "the code looks clean" — produce noisy, unreliable scores. Explicit ones — "the diff modifies exactly one function," "all pre-existing tests pass," "no new dependencies were added" — produce scores you can trust. Spend your effort here. A precise rubric is worth more than a clever scorer.

Scoring: deterministic checks plus LLM-as-judge

Two kinds of scoring cover most agent evals. Deterministic checks are code: run the test suite, parse the diff, assert the output validates against a schema. They're cheap, fast, and unambiguous, and you should use them for everything that can be checked programmatically. But many quality dimensions resist code — was the explanation clear, did the agent follow the spec's intent, is the tone right. For those you use LLM-as-judge: a separate Claude call that reads the agent's output and the rubric and scores each criterion.

The judge is its own design problem. Give it the rubric as explicit, independently gradeable criteria, ask it to score each one separately rather than emit a single vibe-based number, and constrain its output with a schema so you get structured results you can aggregate. Use a capable model as the judge — a weak judge produces a weak signal — and run the judge at a deliberate effort level so it actually reasons through each criterion. Crucially, separate finding from filtering: ask the judge to report every issue it sees with a confidence and severity, then filter downstream. Opus models follow "only report serious problems" instructions literally, which can make a judge silently drop real findings; tell it coverage is the goal and let your aggregation rank.

flowchart TD
  A["Commit / PR opened"] --> B["Run agent on eval task set"]
  B --> C["Deterministic checks: tests, diff, schema"]
  B --> D["LLM-as-judge scores rubric criteria"]
  C --> E["Aggregate to a release score"]
  D --> E
  E --> F{"Score >= quality bar?"}
  F -->|Yes| G["Allow merge / deploy"]
  F -->|No| H["Block release, surface failing criteria"]

Wiring evals into a release gate

An eval suite that runs only when you remember to run it is a suggestion, not a gate. The value comes from putting it in the path of every change. On each pull request or pre-deploy, run the agent across your task set, score with both deterministic checks and the judge, aggregate to a single release score, and compare it to a quality bar. If the score clears the bar, the change ships; if it doesn't, the gate blocks and surfaces exactly which criteria failed on which tasks. That last part matters — a gate that says "score dropped" is annoying; a gate that says "3 tasks now leave failing tests after the prompt change" is actionable.

Evals are also non-latency-sensitive by nature, which makes them a perfect fit for the Message Batches API at half price. Submit the whole task set as a batch, let it run, collect results. You can afford a large, thorough eval set precisely because you're not paying live rates or waiting on a live loop. A broad suite that runs cheaply overnight beats a thin one you run by hand.

Designing a suite that catches real regressions

The trap is an eval set that's too easy or too narrow. If every task is a happy-path case the agent already nails, your suite is green forever and catches nothing. Build the set from real failures: every time the agent misbehaves in production, distill it into a task and add it to the suite. Over time the suite becomes a memory of every way the agent has broken, and a regression on any of them is caught before release. Include adversarial cases — ambiguous specs, tasks that tempt the agent to overreach, inputs designed to trigger the failure modes you've seen.

Re-baseline when you change models. Different Claude versions count tokens differently and behave differently, so a model bump is exactly when you most need the eval loop — run the new model against the full suite, compare scores criterion by criterion, and only promote it if it clears the bar. The eval loop turns a model migration from a leap of faith into a measured decision, which is the entire point of having one.

Frequently asked questions

Do I need a golden answer for every eval task?

No, and for agents you usually can't have one — outputs are open-ended. Grade against an explicit rubric of checkable criteria instead. Deterministic checks (tests pass, diff scope, schema validity) handle the objective parts; an LLM judge scores the subjective ones.

How do I make the LLM judge reliable?

Give it explicit, independently gradeable criteria, have it score each separately with a structured-output schema, use a capable model at a deliberate effort level, and tell it to report every finding with confidence and severity rather than self-filtering — Opus follows "only flag serious issues" literally and may drop real findings.

Where does the eval loop fit in CI?

On every pull request or pre-deploy: run the agent across the task set, score with deterministic checks plus the judge, aggregate to one release score, and block the merge if it's below your quality bar. Run the suite as a batch for the 50% discount on this non-interactive work.

How do I keep the suite from going stale?

Feed it from production failures. Every time the agent breaks, turn that case into an eval task. The suite becomes a growing record of real failure modes, and it must be re-run in full whenever you change models or prompts before promoting the change.

Bringing agentic AI to your phone lines

CallSphere gates its voice and chat agents the same way — rubric-graded evals and a CI quality bar before any release reaches the assistants that answer your calls, use tools mid-conversation, and book work 24/7. See it live at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.