How to Measure Claude Finance Plugins Success

Six weeks into a Claude Cowork rollout, every finance leader hits the same question from their CFO: "Is this actually working?" And the honest answer, for most teams, is "we think so?" — because they never decided up front what working would look like. Vibes are not a metric. If you cannot point to numbers that prove the agentic plugins are earning their keep, you will lose the budget the first time another priority comes along, no matter how impressive the demos felt.

This post is about measuring agentic finance work seriously: the signals that prove a plugin is helping, the ones that warn it's drifting, and how to instrument all of it without turning measurement into a second full-time job.

Key takeaways

• Measure three families: quality (is it right), throughput (is it faster), and trust (do humans need to intervene less over time).
• The single best leading indicator is the human override rate — how often a reviewer changes the agent's proposal.
• Track a small eval set of known-answer cases and run it on a schedule to catch drift before users do.
• Don't only measure speed; a faster wrong answer is worse than a slow right one in finance.
• Tie at least one metric to a business outcome — close-cycle days, reconciliation breaks caught — so the CFO sees value, not activity.

The three families of metrics that matter

Useful measurement in agentic finance breaks into three families, and you need at least one metric from each. Lean on only one and you'll fool yourself.

Quality metrics answer "is the output correct?" The cleanest version is accuracy against a known-answer eval set — a frozen collection of past cases where you already know the right answer. You also watch the material-error rate: how often a wrong number got past review (ideally zero, and tracked seriously when it isn't).

Throughput metrics answer "is this faster or cheaper?" Cycle time per task, number of items a reviewer can process per hour, and token cost per run all live here. These are easy to measure and easy to over-weight — speed is necessary but never sufficient in finance.

Trust metrics answer "are humans needing to step in less?" The headline is the human override rate: of the proposals the agent makes, what fraction does a reviewer change? A falling override rate, with quality holding, is the clearest evidence that a plugin is genuinely maturing.

How the signals flow into a decision

Metrics only matter if they drive an action. Here is how the signals should route into either "scale it up" or "pull it back."

flowchart TD
  A["Plugin run completes"] --> B["Capture: override rate,
cycle time, token cost"]
  B --> C["Weekly eval on known-answer set"]
  C --> D{"Quality holding
& overrides falling?"}
  D -->|Yes| E["Raise autonomy / scope"]
  D -->|No| F{"Quality dropped?"}
  F -->|Yes| G["Pull back & investigate drift"]
  F -->|No| H["Hold; tune spec"]

The decision is never "the demo felt good." It's a gate: quality must hold and overrides must be falling before you grant the plugin more scope or autonomy. If quality drops, you pull back first and investigate second — in finance, the safe direction is always toward more human involvement, not less.

Instrumenting it: a lightweight scorecard

You don't need a data platform to start. A simple per-run record, appended to a table, is enough to compute every metric above. Here is the shape of one row your plugin (or a thin wrapper) can emit on each run:

{
  "run_id": "2026-05-accrual-US-OPCO",
  "task": "accrual_review",
  "items_proposed": 84,
  "items_overridden": 6,
  "override_rate": 0.071,
  "material_errors_escaped": 0,
  "cycle_time_minutes": 41,
  "manual_baseline_minutes": 540,
  "token_cost_usd": 2.18,
  "eval_score": 0.97
}

From a handful of these rows you can chart override rate over time, compare cycle time to the manual baseline, and watch the eval score for drift. The two fields to never lose are material_errors_escaped (your safety signal) and override_rate (your trust signal). Everything else is supporting detail.

Common pitfalls in measuring agentic finance work

• Measuring only speed. A plugin that's 10x faster and occasionally materially wrong is a liability. Always pair throughput with a quality and a safety metric.
• No baseline. "It saved time" is meaningless without the manual number you're comparing against. Record the baseline before you automate.
• Ignoring drift. A plugin that scored well at launch can decay as data, mappings, and models change. Without a scheduled eval, you'll learn about it from a wrong board number.
• Vanity metrics. "Number of runs" or "queries handled" measure activity, not value. Tie at least one metric to a real outcome the CFO cares about.
• Reviewer rubber-stamping. If override rate is zero from day one, your reviewer may be approving blindly. A healthy new plugin shows some overrides that decline as it improves.

Stand up measurement in five steps

1. Before automating, record the manual baseline — time and error rate — for the target task.
2. Build a small frozen eval set of past cases with known-correct answers (20–50 is plenty to start).
3. Emit a per-run scorecard capturing override rate, escaped material errors, cycle time, and token cost.
4. Run the eval on a schedule (each close, or weekly) and chart the score for drift.
5. Set explicit gates: the numbers required to raise autonomy, and the numbers that trigger a pullback.

Which metric answers which question

Question	Metric	Healthy direction
Is it correct?	Eval score / escaped material errors	High / zero
Is it faster?	Cycle time vs. manual baseline	Lower
Do humans trust it more?	Human override rate	Falling (quality held)
Is it affordable?	Token cost per run	Stable or lower

Frequently asked questions

What's the single best metric to start with?

The human override rate. It directly captures whether the agent's work is trustworthy, it's cheap to compute (count the proposals a reviewer changed), and its trend over time tells you whether the plugin is maturing or stalling.

How big should an eval set be?

Start small — 20 to 50 known-answer cases drawn from real past work is enough to catch meaningful drift. Quality and coverage of the cases matter far more than raw count; add cases whenever a new failure mode appears.

How do I show ROI to a CFO?

Pair a throughput metric with a quality metric and tie one to a business outcome they already track — close-cycle days, reconciliation breaks caught, hours redeployed to analysis. Activity counts impress no one; outcomes do.

Measuring agents on the front line, too

CallSphere instruments its voice and chat agents the same way — tracking resolution quality, handoff rates, and outcomes per conversation so you can prove the agent is working, not just busy. See the live metrics in action at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.