Testing and Evals for Parallel Claude Code Agents

You can't ship an agent you can't measure. With a single-shot prompt, eyeballing a few outputs gets you surprisingly far. With parallel Claude Code agents that take dozens of actions, call tools, and coordinate, intuition collapses — a change that fixes one scenario quietly breaks three others, and you won't know until a user hits it. The teams that ship agentic systems confidently all have the same thing: an eval loop that turns "does this feel better?" into a number, and a gate that refuses to release when the number drops. This post is about building that loop for parallel agents.

Key takeaways

• Evals are how you make agent quality measurable; without them every change is a guess.
• Build a scenario set from real tasks and past failures, not toy examples.
• Grade on outcomes (did the task succeed) and trajectory (did it use the right tools efficiently), not just final text.
• Use programmatic checks where you can and an LLM judge where you must — and validate the judge.
• Gate releases in CI: a change only ships if the eval score holds or improves.
• Track cost and latency per scenario alongside quality so you don't optimize one and wreck another.

What an agent eval actually measures

An eval for a parallel-agent system has to capture more than a single-turn LLM eval does. The agent's job spans many steps, so quality has at least three dimensions: outcome (did the end state match what was wanted — tests pass, file correct, ticket closed), trajectory (did it get there efficiently, with the right tools and without loops), and cost (how many tokens and how much wall-clock time). A run that produces the right answer after forty wasteful turns is not a passing run; it's a warning.

An eval is a scored test that runs a fixed scenario through your agent and compares the result to an expected outcome using one or more graders. The art is in choosing scenarios that represent your real distribution of work and graders that are objective enough to trust.

Building a scenario set that catches real bugs

The most common eval mistake is testing the happy path. Your scenario set should be dominated by the cases that have actually gone wrong: the ambiguous request, the missing file, the tool that returns an error, the task that requires the orchestrator to coordinate three workers. Every time a parallel run fails in the wild, distill it into a reproducible scenario and add it to the set. Over time the eval becomes a memory of every mistake your agents have made, which is exactly what stops regressions.

flowchart TD
  A["Code or prompt change"] --> B["Run scenario set"]
  B --> C["Per scenario: outcome grader"]
  C --> D["Per scenario: trajectory + cost check"]
  D --> E{"Score >= baseline?"}
  E -->|No| F["Block release & show regressions"]
  E -->|Yes| G{"Cost & latency ok?"}
  G -->|No| F
  G -->|Yes| H["Promote & update baseline"]

The loop is the point: every change runs the full set, and nothing ships unless the aggregate score holds and the regressions list is empty. The eval set grows as failures are discovered, so the gate gets stricter over time rather than rotting.

Graders: programmatic first, LLM judge second

Prefer deterministic, programmatic grading whenever the outcome is checkable: did the tests pass, does the file match a golden output, did the function return the right value, was the right tool called. These graders are fast, free, and not subject to the same failure modes as the system under test.

// Outcome grader (deterministic)
def grade(scenario, result):
    checks = [
        result.tests_passed == True,
        "DELETE" not in result.sql,          // safety check
        result.files_changed <= scenario.max_files,
    ]
    return all(checks)

// Trajectory grader
def grade_trajectory(trace):
    return trace.tool_calls <= 12 and not trace.had_loop

When quality is subjective — was the explanation clear, was the code idiomatic — use an LLM as a judge. But a judge is itself a model and can be wrong, so validate it: hand-label a sample, check that the judge agrees with humans, and keep its rubric narrow and concrete. A vague "rate this 1–10" judge produces noise; a judge asked specific yes/no questions produces signal.

Make parallel runs reproducible enough to grade

Agents are nondeterministic, which makes naive evals flaky: the same scenario passes one run and fails the next, and you can't tell whether a change helped or you just got lucky. You don't need bit-for-bit determinism, but you do need enough stability to trust a verdict. Pin the model version per scenario, fix any seeds your tools expose, and freeze external dependencies behind recorded fixtures so a flaky API doesn't masquerade as an agent regression.

Then handle the residual variance statistically rather than pretending it's gone. Run each scenario a small number of times and treat a scenario as failing only if it fails consistently, not on a single unlucky sample. This separates real regressions from noise and keeps your gate from blocking good changes over a one-off fluke. Record the pass rate per scenario, not just pass/fail, so a scenario that drifts from passing nine times in ten to six times in ten shows up as a warning before it becomes an outright failure.

Gating releases in CI

Evals only protect you if they block bad changes automatically. Wire the scenario set into your pipeline so that any change to prompts, tools, orchestration, or model selection triggers a full run. The gate is simple: compute the aggregate score and the per-scenario diff against the current baseline; if any scenario regresses or the aggregate drops below threshold, fail the build and print exactly which scenarios broke. This is the difference between an eval suite that's a nice dashboard and one that actually prevents incidents.

Stand up an eval loop in 6 steps

1. Collect 15–30 scenarios from real tasks and past failures, with expected outcomes.
2. Write programmatic outcome graders for everything objectively checkable.
3. Add a trajectory check for tool-call count and loop detection, and record cost and latency.
4. Add a narrow, validated LLM judge only for the subjective dimensions you can't grade in code.
5. Set a baseline from your current agent and wire the suite into CI.
6. On every change, fail the build on any regression; when a change wins, promote it and update the baseline.

Common pitfalls

• Only testing the happy path. Real bugs live in ambiguity and errors; seed the set from actual failures.
• Grading final text only. A correct answer reached via a loop is a latent failure; grade trajectory and cost too.
• Trusting an unvalidated LLM judge. A judge can be confidently wrong; check its agreement with human labels before relying on it.
• Evals that don't gate. A suite you run manually and ignore protects nothing; make it block releases in CI.
• Letting the set go stale. If you stop adding scenarios after each incident, the gate slowly loses its teeth.

Frequently asked questions

What is an eval for an AI agent?

An eval is a scored test that runs a fixed scenario through the agent and compares the result to an expected outcome using one or more graders. For agents it measures outcome, trajectory, and cost together, because a correct result reached inefficiently is still a problem.

Should I use an LLM to grade my agent?

Use deterministic, programmatic grading wherever the outcome is objectively checkable, and reserve an LLM judge for subjective quality dimensions. Always validate the judge against human labels and keep its rubric specific, since a model grading a model can be wrong in correlated ways.

How many scenarios do I need to start?

Start with 15–30 scenarios drawn from real tasks and past failures rather than synthetic examples, then grow the set every time a run fails in the wild. Coverage of your real failure distribution matters far more than raw count.

How do evals gate a release?

Wire the scenario set into CI so any change to prompts, tools, orchestration, or model selection runs the full suite. If the aggregate score drops below baseline or any scenario regresses, the build fails and prints exactly which scenarios broke.

Bringing agentic AI to your phone lines

CallSphere gates its own voice and chat agents with the same eval discipline — real scenarios, objective graders, and a release gate — so every call and message is handled reliably. See it live at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.