How to measure if agentic coding is actually working

A few months into adopting Claude Code, every engineering leader faces the same question from someone holding a budget: is this actually working? The honest, uncomfortable answer is that most teams cannot tell, because they are measuring the wrong things. They count tokens consumed or pull requests opened and call it adoption. Those numbers go up whether the agent is creating value or quietly creating cleanup work. Measuring agentic coding well means tracking signals that distinguish real leverage from expensive motion — and being ruthless about ignoring the metrics that only feel like progress.

Why the obvious metrics lie

The seductive metrics are the ones that move the moment you turn the tool on. Lines of code generated, number of agent sessions, tokens spent, pull requests opened per week — all of these climb with usage and tell you almost nothing about value. An agent can generate ten thousand lines that a human spends a day deleting. It can open twenty pull requests, fifteen of which get reverted. High activity with low yield is the default failure mode of badly measured agentic adoption, and the vanity metrics actively hide it.

The deeper problem is that these metrics measure the agent's output, when what you care about is the team's outcome. The right frame is not "how much did the agent do" but "did the system of humans-plus-agents ship more valuable, correct work per unit of human effort than before." That reframing changes everything you choose to count.

The metrics that actually correlate with value

Four families of signal hold up. The first is throughput with a quality gate: not raw PR count, but the number of changes that ship and stay shipped — merged, not reverted within a week, no follow-up incident. A change that ships and survives is real output; one that ships and gets rolled back is negative output dressed as progress.

The second is rework rate: what fraction of agent-generated changes need substantial human rewriting before they are acceptable. A low and falling rework rate means your specs, context, and tooling are good and the agent is genuinely carrying load. A high rework rate means the agent is generating drafts you mostly throw away, which can be slower than writing it yourself.

The third is human review load per shipped change. Agentic coding shifts effort from writing to reviewing, so if review time per change is exploding, your net leverage may be negative even as output rises. The fourth is cycle time: the wall-clock from a problem being identified to its fix being live in production. This is the metric customers actually feel, and it captures the whole human-plus-agent system rather than any one part.

flowchart TD
  A["Agentic change produced"] --> B{"Merged & survived 1 week?"}
  B -->|No, reverted| C["Count as negative — investigate cause"]
  B -->|Yes| D{"Needed major human rewrite?"}
  D -->|Yes| E["High rework — tighten specs & tools"]
  D -->|No| F["Measure review time per change"]
  F --> G{"Review load rising fast?"}
  G -->|Yes| H["Net leverage may be negative"]
  G -->|No| I["Real leverage — cycle time should drop"]

Quality and safety signals you cannot skip

Throughput without quality is how teams ship faster straight into more incidents. So any honest measurement program tracks the safety side too. Watch your change failure rate — the fraction of deploys that cause an incident or require a hotfix — and watch it specifically for agent-heavy changes versus human-written ones. If the agentic changes fail more often, your verification is too thin and the throughput gain is borrowed against future firefighting.

A second safety signal is escaped-defect rate: bugs that pass your tests and reviews and reach users. Agents introduce a characteristic class of plausible-but-wrong changes that shallow tests miss, so a rising escaped-defect rate is an early warning that your eval gates are not catching what the agent gets wrong. The fix is rarely "use the agent less" — it is "strengthen the evals" — but you only know to do that if you measure it.

Leading indicators that predict trouble early

Outcome metrics like cycle time and failure rate are lagging — by the time they move, the cause is weeks old. The teams that manage agentic adoption well also watch leading indicators. One is the diff-to-accept ratio: how much of what the agent proposes survives review unchanged. A falling ratio means your context and specs are degrading before the lagging metrics catch up. Another is retry and loop frequency: agents that need many iterations to converge on a task signal an ambiguous spec or a missing tool, and that friction predicts both wasted tokens and lower-quality output.

A third, softer signal is engineer trust, which you can survey for directly: do your engineers trust the agent's output enough to review it efficiently, or are they re-reading every line out of fear? Distrust shows up as ballooning review time long before it shows up in cycle time, and it usually means the verification infrastructure — the tests and evals people rely on — is not yet strong enough to delegate to.

Putting it together without drowning in dashboards

You do not need all of these instrumented on day one. A workable starting set is four numbers: surviving-change throughput, rework rate, review load per change, and change failure rate for agentic work. Track them as a trend, compare agent-heavy work against the team's baseline, and resist the urge to celebrate the vanity metrics that move first. If surviving throughput rises while rework, review load, and failure rate stay flat or fall, agentic coding is genuinely working. If throughput rises but the others climb with it, you are buying speed with debt — and now you can see it, which is the entire point of measuring at all.

Frequently asked questions

What is the single best metric for agentic coding ROI?

If you can track only one, track surviving-change throughput — changes that merge and stay merged without a revert or incident — normalized by human effort. It captures both that work shipped and that it was good enough to last, which is exactly the combination vanity metrics miss.

How do I separate the agent's impact from everything else changing?

Compare agent-heavy work against your own historical baseline and against human-written changes in the same period, rather than against an industry number. The cleanest read comes from looking at the trend within your team before and after adoption, controlling for obvious confounders like team size or a major refactor.

Are tokens spent a useful cost metric?

As a pure cost line, yes — token spend is a real bill, especially for multi-agent workflows that use several times more tokens than single-agent ones. As a value metric it is meaningless. Pair token cost with surviving throughput to get cost-per-shipped-change, which is the number worth managing.

How quickly should I expect metrics to improve?

Expect rework rate and review load to be ugly at first and improve over one to three months as your specs, MCP tooling, and evals mature. If those numbers are not improving after a quarter of real investment, the problem is usually thin verification infrastructure, not the model.

Measuring agents on the phone, too

CallSphere instruments its voice and chat agents the same way — resolution rate, escalation rate, and booked outcomes per conversation, not vanity call counts. See the metrics that prove agentic value at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.