Testing and Evals for Claude Code Threat Agents

You change one line of your detection agent's system prompt to handle a new alert type, ship it, and a week later discover it now mislabels a category of real intrusions as benign. Nobody noticed because the agent is non-deterministic and there was no test that would have caught the regression. This is the central problem with shipping agentic systems: traditional unit tests don't fit, yet the cost of a quality regression in threat detection is a missed breach. The answer is an eval loop — a repeatable way to measure agent quality and gate every change behind it. This post is about building one that actually protects you.

Why agent quality needs evals, not unit tests

A unit test asserts a deterministic output for a fixed input. An agent given the same alert twice may take different tool paths and phrase its verdict differently while still being correct — or take an identical path and be subtly wrong. So you don't assert exact strings; you measure outcomes across a population of cases. An eval is a structured test that runs an agent against a labeled dataset and scores its outputs against known-correct answers to produce quality metrics. For a triage agent the core metrics are familiar from detection itself: true-positive and false-positive rates, false-negative rate (missed real threats — the most expensive error), and the rate of correct escalation versus auto-close.

The unit of an eval is a case: a realistic alert payload plus a gold label (benign / suspicious / malicious), and ideally the expected action (auto-close, escalate, contain). You build a dataset of these and run the whole agent against all of them on every change.

Build the dataset from your hardest real cases

A good eval set is not a handful of toy examples; it is a curated collection of the cases that actually trip the agent up. Seed it from production: every time an analyst overrides the agent's verdict, capture that alert and its correct label into the eval set. Every confirmed incident becomes a malicious case; every noisy false alarm becomes a benign case the agent must learn to dismiss. Over a few months this turns your real operational pain into your regression suite. Deliberately over-weight the dangerous and ambiguous cases — the near-misses where a wrong call costs the most — because average accuracy on easy alerts tells you nothing about the failures that hurt.

Keep a portion of the set frozen as a held-out benchmark you never tune against, so your headline numbers stay honest, and let the rest grow as new failure modes appear.

flowchart TD
  A["Change to agent prompt/tools"] --> B["Run agent over labeled eval set"]
  B --> C["Score: TP, FP, false-negatives, escalation"]
  C --> D["LLM-judge rates reasoning quality"]
  D --> E{"Metrics >= release gate?"}
  E -->|No: regression| F["Block release, inspect diffs"]
  E -->|Yes| G["Promote to canary"]
  G --> H["Monitor live overrides"]
  H --> I["Add new failures back to eval set"]
  I --> B

Scoring: deterministic checks plus an LLM judge

Score in two layers. First, the deterministic layer: did the verdict label match the gold label? Did the agent take a forbidden action? Did it stay under its tool-call budget? These are exact, cheap, and unambiguous, and they catch the most important regressions. Second, the judgment layer: was the agent's reasoning sound, even when the label was right? A verdict can be correct by luck. Here you use an LLM-as-judge — a separate Claude call given the alert, the agent's reasoning trace, and a rubric, asked to rate whether the reasoning actually supports the verdict.

LLM judges need discipline to be trustworthy. Give the judge a concrete rubric rather than "is this good," use a capable model (Opus 4.8) as the judge even if a cheaper model runs production, and validate the judge against human ratings on a sample so you know it agrees with your analysts. A judge nobody has calibrated is just another opinion. Used well, it scales the qualitative review that a human can't do across thousands of cases.

Gate releases on the numbers

An eval is only protective if it can block a release. Wire the eval run into CI: every change to the prompt, the tools, or the model version triggers a full eval pass, and the pipeline fails if any gate is breached. The most important gate is the false-negative rate — never ship a version that misses more real threats than the current production version, full stop. Set additional gates on false-positive rate (so the agent doesn't drown analysts in noise) and on judge-rated reasoning quality. When a gate trips, the pipeline surfaces the specific cases that regressed so you can see exactly what broke, not just that something did.

Pin the model version in evals. When Anthropic ships a new model, treat it as a change that must pass the same gates before adoption — model upgrades can shift behavior on edge cases, and your eval set is precisely how you catch a regression that a release note wouldn't mention.

Close the loop with canary and production feedback

Passing offline evals earns a version a canary slot, not a full rollout. Run the new version on a slice of live traffic in shadow or low-stakes mode and compare its verdicts against the incumbent and against analyst overrides. The override rate in production is your truest quality signal, because it reflects cases your eval set hasn't seen yet. Feed every override and every confirmed incident back into the eval dataset, and the system compounds: each real-world failure becomes a permanent test that the next version must pass. That loop — measure, gate, canary, learn — is what lets you keep changing a detection agent without quietly breaking it.

Frequently asked questions

What exactly is an eval for an AI agent?

An eval is a structured test that runs the agent against a labeled dataset of realistic cases and scores its outputs against known-correct answers to produce quality metrics — for a threat agent, things like false-negative rate and correct-escalation rate. Unlike a unit test, it measures outcomes across a population rather than asserting one exact output, because agents are non-deterministic.

How do I build an eval dataset for threat detection?

Harvest it from production: every analyst override and every confirmed incident becomes a labeled case, weighted toward the ambiguous and dangerous situations where wrong calls cost most. Keep a frozen held-out portion for honest headline metrics and let the rest grow as new failure modes appear.

Can I trust an LLM as a judge?

Yes, if you discipline it: give it a concrete rubric, use a capable model like Opus 4.8 as the judge, and validate its ratings against human analysts on a sample before relying on it. Use deterministic checks for label correctness and forbidden actions, and reserve the LLM judge for grading reasoning quality at a scale humans can't reach.

How should evals gate a release?

Run the full eval suite in CI on every prompt, tool, or model change, and fail the build if any gate is breached — above all, never ship a version with a worse false-negative rate than production. Pin the model version so model upgrades pass the same gates, then canary the winner on live traffic before full rollout.

Bringing agentic AI to your phone lines

CallSphere gates its voice and chat agents the same way — labeled evals and live override feedback ensure assistants that answer every call and message keep getting more accurate, never less. See it live at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.