How to measure success shipping apps with Claude Code

A non-technical PM finishes their first week with Claude Code and the app does something. Is that success? It might be — or it might be a polished demo that will collapse the first time a real user does something unexpected. The hard part of measuring agentic development is that the obvious signal, "the thing runs," is also the most misleading one. Claude Code is exceptional at producing software that appears to work. Proving it actually works, and that you are building the skill to keep producing working software, requires better metrics.

This matters more for PM-led projects than for traditional teams, because the PM lacks the engineer's reflexive sense of "this feels solid." Without that gut instinct, you need explicit signals to stand in for it. The good news is that the right metrics are concrete and trackable, and most of them are leading indicators — they warn you before things go wrong rather than after. Here is what to measure and what to ignore.

Why "it works" is the wrong primary metric

The demo working tells you the happy path functions once, under conditions you controlled. It says nothing about what happens with malformed input, concurrent users, network failures, or malicious probing. A booking app that books appointments beautifully in the demo can still double-book under load, leak one user's data to another, or fall over when an email fails to send. These are exactly the failures that do not show up in "it works" and exactly the ones that hurt in production.

So the foundational shift is from presence of function to resilience of function. A useful definition: a success metric is a measurable signal that distinguishes a system that genuinely works under real conditions from one that merely appears to work in a demo. Good agentic metrics all share this property — they probe the gap between the demo and reality. The flowchart below shows how to think about which signals matter at which stage.

flowchart TD
  A["PM ships a feature"] --> B["Leading signal: tests pass & cover edge cases?"]
  B -->|No| C["Not ready: add coverage"]
  B -->|Yes| D["Process signal: did PM review the diff?"]
  D -->|No| C
  D -->|Yes| E["Ship to small group"]
  E --> F["Outcome signal: real-user errors & task completion"]
  F --> G{"Metrics healthy?"}
  G -->|No| H["Diagnose & iterate"]
  G -->|Yes| I["Expand rollout"]
  H --> A

Leading indicators: the signals that warn you early

The most valuable metrics are the ones you can read before users ever touch the app. The first is test coverage of edge cases. Not just "are there tests" but "do the tests cover the unhappy paths" — invalid input, unauthorized access, concurrent operations, failure of external services. A PM can track this concretely by maintaining a checklist per feature and confirming Claude Code wrote and passed tests for each case. A feature with happy-path tests only is a feature that will surprise you.

The second leading indicator is review depth: the percentage of agent-generated changes the PM actually read and understood before accepting. This is a process metric about your own discipline. If you are accepting code you did not review, the number is zero and your risk is high, regardless of how the app looks. Tracking it honestly — even just "did I read and reason about this diff: yes/no" — keeps the verification habit alive when deadline pressure tempts you to rubber-stamp.

The third is defect-catch timing. Where are you finding bugs — in your own testing, in the guardian's review, or from real users in production? A healthy project catches most defects in the first two stages. If users are routinely the ones discovering problems, your earlier signals are too weak and you should strengthen testing and review before shipping more.

Outcome metrics: proof from real users

Once the app is live, the metrics shift to what users experience. Task completion rate is the most honest: of the people who started the core action — booking an appointment, submitting a form, completing checkout — what fraction finished successfully? A high completion rate is strong evidence the app genuinely works; a low one points to friction or breakage the demo hid. Pair it with error rate: how often does a user action result in an error or a broken state? Even a small but steady error rate signals an edge case you have not handled.

Two more outcome signals matter. Time-to-recovery measures how long it takes you to fix and redeploy when something does break — a proxy for whether you truly understand your own app. PMs who built real literacy recover in minutes; those who shipped code they cannot read are stuck for hours. And cost per outcome guards against the runaway-spending failure mode: track what each booking or transaction costs in paid services, and watch for the agent-written loop that quietly inflates it.

Metrics that mislead, and how to avoid them

Some numbers feel like progress but measure nothing useful. Lines of code generated is meaningless — Claude Code can produce thousands of lines in minutes, and volume correlates with nothing good. Features shipped per week is a vanity metric if those features are untested; shipping fragile features fast is how PM projects accumulate the scope-induced fragility that eventually breaks everything. Velocity of the agent tempts you to measure how fast Claude Code works, when what matters is how fast you can verify what it produced.

The trap underneath all of these is measuring activity instead of outcome. The agent is fast by default; that is not your achievement and not your risk. Your achievement is shipping software that holds up, and your risk is shipping software that does not. Keep your dashboard pointed at resilience and real-user outcomes, and treat any metric that goes up just because the agent typed more as noise. A simple weekly review — coverage, review depth, defect-catch timing, completion rate, error rate, cost per outcome — gives a non-technical PM the instrument panel an engineer carries in their head.

Frequently asked questions

What is the single best metric for a non-technical PM?

Task completion rate once live, and edge-case test coverage before that. Together they answer the only question that matters: does this genuinely work for real users under real conditions, or does it just look like it does in a demo?

How do I measure something as fuzzy as review depth?

Keep it simple and honest: for each accepted change, did you read it and could you explain what it does? Even a yes/no log keeps you disciplined. The metric exists to catch the moment deadline pressure tempts you to accept code you have not understood.

Should I track how fast Claude Code works?

No. Agent speed is high by default and tells you nothing about quality. What constrains your project is how fast you can specify and verify, not how fast the agent types. Measuring agent velocity points your attention at the wrong bottleneck.

When do I know the app is actually ready to ship?

When edge-case tests pass, you have reviewed the risky code, a guardian has checked anything security-sensitive, and a small real-user trial shows a high completion rate and low error rate. "It works in the demo" is the start of that checklist, not the end.

Measuring agentic outcomes on your phone lines

The same outcome-first measurement — completion rates, error rates, cost per result — is how CallSphere proves its agents work on voice and chat. Our assistants answer every call and message and book work 24/7, measured by real outcomes, not activity. See the signals live at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.