How to Measure LLM Source-Code Security That Works

A security leader can stand up Claude as a code reviewer in an afternoon. Proving it actually makes the codebase safer takes a measurement practice — and most teams never build one. They watch the agent post findings, feel reassured, and call it a win. Six months later, when an auditor or a board member asks 'how do you know this is working?', they have anecdotes and no evidence. This post is about the metrics and signals that turn an LLM code-security program from a vibe into something you can defend, tune, and trust.

The hard part is that the obvious metric — number of findings — is almost useless and sometimes actively misleading. A spike in findings might mean the agent got sharper or it might mean it started hallucinating. A drop might mean the codebase got safer or that the agent quietly regressed after a prompt change. To measure well, you have to measure the things that are genuinely hard to game: precision, the cost of being wrong, and the rate at which real risk actually leaves the codebase.

Precision and recall are the foundation

Every serious program starts by measuring whether the agent's findings are true. Precision — the share of Claude's findings that turn out to be real, exploitable issues — is the metric your developers feel most directly, because low precision means wasted time and eroded trust. You measure it by having humans adjudicate a sample of findings and recording the true-positive rate. If precision is low, developers learn to ignore the agent, and the whole program dies of crying wolf.

Recall — the share of real vulnerabilities the agent actually catches — is harder to measure because it requires knowing the ground truth, which is precisely what you don't have. The practical proxy is a curated benchmark: a fixed set of repositories with known, labeled vulnerabilities (including ones you've deliberately reintroduced) that you run the agent against on every meaningful change to its prompt, tools, or model. That benchmark turns recall from an unknowable into a tracked number.

Build an eval loop before you build the dashboard

The single most important investment is an evaluation suite that gates changes to your security agent. Without it, every prompt tweak is a leap of faith. With it, you can ask: did this change to the review prompt improve precision without hurting recall? Did upgrading the model help? Did adding a Skill that encodes our crypto standards actually reduce missed crypto issues? The eval loop converts those questions from arguments into experiments.

flowchart TD
  A["Change: prompt, Skill, or model"] --> B["Run against labeled benchmark repos"]
  B --> C{"Precision & recall vs baseline?"}
  C -->|Regressed| D["Block the change"]
  C -->|Improved or flat| E["Promote to production agent"]
  E --> F["Sample live findings, humans adjudicate"]
  F --> G["Track precision drift over time"]
  G --> H{"Drift detected?"}
  H -->|Yes| A
  H -->|No| I["Continue monitoring"]

The benchmark should include both positive cases (known vulnerabilities the agent must find) and negative cases (clean code the agent must not flag). Negative cases matter as much as positive ones, because a model that flags everything has perfect recall and worthless precision. Run the suite on a schedule and on every change, and treat a regression as a blocker, exactly as you'd treat a failing test.

Outcome metrics: is risk actually leaving the codebase?

Precision and recall tell you the agent is accurate; outcome metrics tell you the program is working. The signal that matters most is mean time to remediate for issues the agent surfaces — if Claude finds real problems but nothing gets fixed faster, you've added a reviewer, not a control. Track the time from finding to merged fix, and watch whether it shrinks as the agent starts proposing patches alongside findings.

A second outcome metric is vulnerability recurrence: how often the same class of issue reappears after you've fixed it. A healthy program drives recurrence toward zero because findings get converted into regression tests and Agent Skills that catch the pattern's return. Rising recurrence is a sign your fixes aren't sticking and your institutional learning isn't being captured. Pair this with coverage — the share of pull requests and code paths the agent actually reviews — so you know your numbers reflect the real codebase, not a corner of it.

Trust signals from the humans in the loop

Some of the most predictive metrics are behavioral. Track the developer accept rate on the agent's findings and proposed patches — when developers consistently accept and act on them, the agent has earned trust; when accept rates fall, something has regressed even if your benchmark hasn't caught it yet. Watch the override rate at the merge gate, where humans correct the agent. A healthy override rate is non-zero (humans are still adding judgment) but stable; a sudden rise signals the agent is drifting.

It is worth instrumenting the cost side too. Multi-agent security reviews can use several times more tokens than a single-agent pass, so track cost per reviewed pull request and per confirmed finding. A program that finds real bugs but at runaway cost will get cut; one that can show cost-per-confirmed-finding trending down as Skills and prompts improve is one leaders keep funding.

Reporting that survives an audit

Finally, package these into a report an auditor or executive can read. The strongest version pairs accuracy metrics (precision against an adjudicated sample, recall against the benchmark) with outcome metrics (time to remediate, recurrence, coverage) and a clear statement of the human controls — that the agent proposes and humans gate every consequential merge. That combination shows not just that you deployed an AI reviewer, but that you know how well it performs and how it's governed.

To define it cleanly: measuring an LLM code-security program means continuously tracking the accuracy of the agent's findings, the speed and durability of remediation, and the trust signals from the humans supervising it — gated by an eval suite that blocks regressions before they reach production. Without that loop, you have a tool. With it, you have a control you can prove.

Frequently asked questions

Isn't the number of findings a good measure of success?

No — it's one of the most misleading metrics available. More findings can mean a sharper agent or a hallucinating one; fewer can mean a safer codebase or a regressed agent. Measure precision (are findings real?), recall against a labeled benchmark, and outcome metrics like time to remediate instead.

How do I measure recall when I don't know what I'm missing?

Build a curated benchmark of repositories with known, labeled vulnerabilities — including ones you deliberately reintroduce — and run the agent against it on every change. The benchmark gives you a fixed ground truth, turning recall from an unknowable into a tracked number you can defend.

What's the most important thing to build first?

An evaluation suite that gates changes to the agent. Before any dashboard, you need to know whether a prompt tweak, a new Skill, or a model upgrade improved or regressed precision and recall. The eval loop turns every change from a leap of faith into a measured experiment.

Which metric best predicts whether the program will survive?

Developer accept rate combined with cost per confirmed finding. If developers trust and act on findings while cost trends down as your prompts and Skills improve, the program demonstrates clear value. Falling accept rates or runaway token cost are the early signals that it's in trouble.

Bringing measurable agentic AI to your phone lines

CallSphere instruments its voice and chat agents the same way — precision, resolution rate, and human-override signals on every call — so you can prove the AI is working, not just assume it. See it live at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.