How to Measure Claude Computer Use Success

The fastest way to fool yourself about a computer-use deployment is to watch it succeed. Claude logs in, navigates, fills the form, submits, and you nod — it works. But 'it worked once while I watched' is not a metric, and a capability that operates a real cursor on real software needs to be measured the way you would measure a new employee you are about to leave unsupervised: not by whether they can do the task, but by how often, how cheaply, how safely, and how independently they do it across hundreds of runs you did not watch. Getting these measurements right is the difference between a deployment you can defend and one that quietly accumulates silent errors until something expensive breaks.

Why task-completion rate is a misleading headline

Task-completion rate — did the agent finish the task — is the number everyone reports and the number that hides the most. It treats a run that finished correctly and a run that finished with a wrong value entered as the same outcome, because both 'completed.' For computer use, completion without correctness is worse than failure: a clean failure stops and flags; a confident wrong completion writes bad data into a live system and moves on. So the first principle of measuring computer use is to separate completion from correctness, and to treat correctness as the metric that actually matters.

Measuring correctness requires ground truth — a set of tasks where you know the right outcome independently of what the agent did. That is what an eval set is: tasks with known-correct results you can replay. Without it, you are grading the agent on its own answer key. With it, you get a correctness rate that does not lie, and you can watch it move every time you change a prompt or a model.

A practical wrinkle: correctness for computer use is often partial, not binary. A run might enter two of three fields perfectly and fumble the third, and collapsing that into a single pass/fail throws away the information you most need. The better practice is field-level or step-level scoring — grade each sub-outcome against ground truth so your correctness rate reflects what fraction of the work was right, not just whether the whole run cleared a bar. This granularity also tells you which part of the task is fragile, which is exactly the input your engineering effort should be aimed at.

The four metrics that actually matter

A defensible computer-use scorecard rests on four numbers, each answering a different question. Together they tell you whether the deployment is working, improving, and safe to trust further.

flowchart TD
  A["Run completes"] --> B["Compare to eval ground truth"]
  B --> C{"Output correct?"}
  C -->|Yes| D["Correctness rate ++"]
  C -->|No| E["Error logged + categorized"]
  A --> F["Count human interventions"]
  A --> G["Sum tokens + time = cost/task"]
  E --> H{"Reversible?"}
  H -->|No| I["Critical incident signal"]
  H -->|Yes| J["Recoverable error signal"]
  D --> K["Scorecard"]
  F --> K
  G --> K
  I --> K

Correctness rate against an eval set is the headline. Intervention rate — how often a human has to step in, correct, or rescue a run — is your autonomy gauge; it should fall over time, and if it stalls high, the workflow is not ready to run unattended. Cost per successful task (tokens plus wall-clock time, divided by correct completions) is the number that decides whether the automation is actually cheaper than the human it replaced; a run that takes many screenshots and retries can quietly cost more than a contractor. And the irreversible-error rate is the safety number that should be zero and that you alarm on the instant it is not.

Leading signals, not just lagging numbers

The four metrics above are lagging — they tell you what already happened. The teams that catch problems early also watch leading signals inside the run. A rising number of steps per task often means Claude is getting confused and wandering; a sudden change in the reasoning trace's tone or goal can indicate a misread or an injection attempt; a climbing retry count is an early warning that the target software changed under the agent. These signals move before the lagging metrics do, which is exactly why they are worth instrumenting.

The practical move is to log, on every step, a screenshot, the chosen action, and the reasoning, then aggregate per-run statistics: step count, retry count, and time to completion. When any of those drifts from its baseline, investigate before the correctness rate drops. A layout change in the target app, for example, shows up as more retries days before it shows up as wrong outputs.

There is a second class of signal that lives in the human layer rather than the agent: the reasons humans intervene. Logging not just how often a person stepped in but why — ambiguous match, unexpected modal, low-confidence read, suspicious trace — turns the intervention rate from a single blunt number into a map of exactly where the workflow is weakest. If half of all interventions trace to one screen, that screen is your next engineering target, and fixing it moves the autonomy needle more than any prompt tweak. Categorized interventions are the cheapest roadmap you will ever get for hardening a computer-use deployment.

Setting thresholds and gates

Metrics without thresholds are decoration. Decide in advance what each number must be for the agent to keep its current level of autonomy. A common shape: correctness above your bar on the eval set, intervention rate trending down, irreversible-error rate at exactly zero, and cost per task below the human baseline. Wire these as gates so the system itself enforces them — if the eval correctness drops below threshold after a model or prompt change, the deployment falls back to shadow mode automatically rather than continuing to run autonomously on a regression.

Metric	Question it answers	Bad sign
Correctness rate	Is the output right?	Drops after a change
Intervention rate	Can it run unattended?	Stays high / rises
Cost per successful task	Is it actually cheaper?	Above human baseline
Irreversible-error rate	Is it safe?	Anything above zero

Key takeaways

• Separate completion from correctness; measure correctness against an eval set with known-good outcomes.
• Track four numbers: correctness rate, intervention rate, cost per successful task, and irreversible-error rate.
• Watch leading signals (steps per task, retries, trace drift) to catch problems before the lagging metrics move.
• Set thresholds in advance and gate autonomy on them; a regression should auto-fall-back to shadow mode.
• Cost per successful task — not raw token cost — decides whether the automation actually pays for itself.

Instrument it in five steps

1. Build an eval set of real tasks with independently known-correct outcomes; start with 20–40.
2. Log screenshot, action, and reasoning on every step; aggregate step count, retries, and time per run.
3. Compute the four core metrics per run and roll them up daily.
4. Set explicit thresholds for each, including a zero bar on irreversible errors.
5. Wire an automatic fallback to shadow mode when correctness drops below threshold after any change.

Common pitfalls

• Reporting completion as if it were correctness. A finished run with a wrong value is a failure that lies. Measure correctness against ground truth.
• Ignoring cost per task. Many screenshots and retries can make an automation cost more than the human. Track cost per successful task, not per run.
• No leading signals. If you only watch lagging metrics, you learn about regressions after they ship bad data. Instrument step and retry counts.
• Thresholds set after the fact. Deciding what 'good' means after you see the numbers invites rationalization. Set gates before you deploy.
• Tolerating a nonzero irreversible-error rate. 'Only one bad wire transfer' is not an acceptable rate. Alarm on the first one.

Frequently asked questions

What is the single best metric for computer use?

Correctness rate against an eval set, because it is the only one grounded in independent truth. Intervention rate and cost matter for operations, and irreversible-error rate matters for safety, but correctness is the foundation everything else sits on.

How do I measure correctness without ground truth?

You cannot, reliably — which is why building an eval set is step one. Take real historical tasks where you know the right outcome, replay them, and compare. Even 20 cases give you a number that moves meaningfully when the agent does.

Why track cost per successful task instead of total cost?

Because failed and rescued runs still consume tokens and time. Dividing by successful completions tells you the true unit economics — what each good outcome actually costs — which is the number you compare against the human baseline.

What should trigger an automatic rollback?

Any eval correctness drop below your threshold after a model or prompt change, and any irreversible error at all. Both should drop the deployment to shadow mode automatically so a regression cannot keep running autonomously while no one is looking.

Bringing agentic AI to your phone lines

CallSphere measures its voice and chat agents the same way — correctness against real outcomes, intervention rate, and cost per booked job — so every conversation that gets handled unattended has earned the trust. See the metrics in action at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.