Testing and Evals for Claude Security Agents

You can hand-test a Claude security agent a dozen times, watch it correctly summarize a vulnerability scan, and ship it — and then discover in production that it silently mislabels a critical finding as informational one run in twenty. Agents are non-deterministic, security work is high-stakes, and "it worked when I tried it" is not a quality bar. The only thing that turns an impressive demo into a dependable system is a real eval loop.

This post is about how to test and evaluate Claude agents that connect to security and compliance tools: what to measure, how to build graders you trust, and how to gate releases so a regression cannot reach production unnoticed.

What you are actually measuring

Start by naming the failure you care about. For a security agent, raw "accuracy" is too blunt. Break quality into dimensions: did the agent call the right tools in a reasonable order (trajectory quality), did it produce the correct conclusion (outcome quality), did it avoid forbidden actions like triggering a remediation it was not allowed to (safety), and did it stay within cost and latency budgets (efficiency). A run can ace the answer and still fail safety, and you need to catch both.

The most important early decision is to evaluate outcomes, not exact wording. A scan summary can be phrased a hundred correct ways, so grading on string match is hopeless. Instead, assert on structured claims: the finding count by severity, the specific CVEs flagged, the recommended action's category. Make the agent emit a structured result alongside its prose so you have something checkable.

Build a fixture set from real and adversarial cases

An eval is only as good as its dataset. Build a fixture set that captures three kinds of cases. Golden cases: real scan outputs and log bundles with a known-correct expected conclusion. Regression cases: every bug you have ever fixed, frozen as a test so it cannot return. And adversarial cases: inputs containing prompt-injection strings, malformed tool responses, and ambiguous findings that should trigger a request for clarification rather than a confident guess.

flowchart TD
  A["Code / prompt change"] --> B["Run agent over fixture set"]
  B --> C["Collect trajectory + structured output"]
  C --> D["Graders: outcome, trajectory, safety, cost"]
  D --> E{"Scores meet thresholds?"}
  E -->|Yes| F["Gate passes — allow merge / deploy"]
  E -->|No| G["Block release & surface failing cases"]
  G --> A

Mock the tools in your fixtures. You do not want your eval suite hitting a live SIEM — it would be slow, expensive, and non-reproducible. Record real tool responses once and replay them, so the same input produces the same tool data every time and you are testing the agent's reasoning, not the network. An eval loop is the repeatable cycle of running an agent against a fixed dataset, scoring its behavior with graders, and gating releases on those scores.

Choosing graders you can trust

You have three grader types and should use all three. Programmatic graders are exact and cheap — assert the structured finding count matches, assert no forbidden tool was called, assert the run stayed under a token budget. Use these wherever the property is checkable. LLM-as-judge graders handle the fuzzy parts: was the written summary faithful to the evidence, was the recommended remediation reasonable. Use a capable model like Claude as the judge, give it a precise rubric, and validate the judge against human labels before you trust it. Human review stays in the loop for a sampled slice, especially on safety-critical cases.

A subtlety teams miss: grade the trajectory, not just the final answer. An agent that reaches the right conclusion by calling a destructive tool it should never touch passed on outcome but failed on safety. Your safety grader should inspect the full tool-call sequence and fail any run that invoked a forbidden action, regardless of how the answer turned out.

Gate releases in CI

An eval that you run manually when you remember is theater. Wire it into the release path. On every prompt change, tool-definition change, or model upgrade, run the full fixture suite, compute scores per dimension, and block the merge or deploy if any score falls below threshold or regresses against the previous baseline. This is the single mechanism that lets you change prompts and swap models with confidence instead of dread.

Model upgrades are where this pays off most visibly. When a new Claude version ships, you do not guess whether it is better for your agent — you run your eval suite on both and compare scores across outcome, safety, and cost. Maybe the newer model is more accurate but more expensive on this task, and you route accordingly. The eval suite turns a model migration from a leap of faith into a measured decision.

Watch for eval rot

Evals decay. As you fix bugs by adding the failing case to the suite, the agent can start to overfit to your fixtures while real-world inputs drift away from them. Counter this by continuously sampling production runs, having humans label a slice, and feeding fresh and surprising cases back into the fixture set. Track not just the pass rate but the distribution of failures over time — a rising rate of "clarification needed" cases, for instance, may signal that your inputs have shifted and the suite needs new examples.

Frequently asked questions

What should I grade in a security agent eval?

Four dimensions: outcome (correct conclusion), trajectory (right tools in a sensible order), safety (no forbidden or destructive actions), and efficiency (within token and latency budgets). A run can pass the answer and still fail safety, so grade all four.

How do I evaluate non-deterministic agent output?

Assert on structured claims, not exact text. Have the agent emit a checkable result — finding counts by severity, flagged CVEs, action category — and grade those programmatically, reserving LLM-as-judge for the fuzzy parts like summary faithfulness.

Should my evals call live security tools?

No. Record real tool responses once and replay them as mocks so runs are fast, cheap, and reproducible. You want to test the agent's reasoning against fixed data, not the reliability of a live SIEM connection.

How do evals help with Claude model upgrades?

They turn a migration into a measurement. Run your fixture suite on both the old and new model, compare scores across outcome, safety, and cost, and decide based on data instead of vibes — including routing different steps to different models.

Tested agents on every call

CallSphere gates its voice and chat agents behind the same kind of eval loop, so quality is measured before any change reaches a live conversation. See the results at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.