Testing and evals for Claude agents in finance

"It worked when I tried it" is not a release criterion for an agent that touches money. Yet that is roughly the bar a lot of teams ship against, because evaluating an agent feels harder than evaluating ordinary software. There is no single expected output to assert against; the same financial question can have several acceptable answers, the model is non-deterministic, and quality lives in a hundred edge cases you didn't think to check. The teams that ship Claude agents into financial workflows with confidence all have one thing in common: a real eval loop that turns vague "seems good" judgments into numbers, and gates releases on those numbers.

An eval is not a unit test, though it borrows the discipline. A unit test asserts an exact return value. An eval scores a behavior against a rubric across many examples and reports a distribution. The goal is not to prove the agent is perfect — it never will be — but to measure whether a change made it better or worse, and to refuse releases that regress. This post lays out how to build that loop for a financial agent, from datasets to scoring to gating.

Start with a dataset that looks like production

Every eval loop begins with a dataset of realistic scenarios. For a financial agent that means real-shaped tasks: a reconciliation with a deliberate discrepancy, a transaction the agent should flag as suspicious, a customer query that requires looking up the right account, an ambiguous request the agent should ask to clarify rather than guess. Pull these from production logs once you have them — anonymized and with sensitive values masked — because real user behavior is stranger and more varied than anything you'll invent at a whiteboard.

Crucially, your dataset must include the failure cases you've already hit. Every time the agent does something wrong in production, that scenario becomes a permanent eval case. This is how an eval suite earns its keep: it ratchets. A bug can be fixed once and then guarded forever, so the agent never silently regresses on a mistake it made before. A useful definition: an agent eval is a repeatable measurement of agent behavior against a fixed dataset and a scoring rubric, run on every candidate change to detect quality improvements and regressions before release.

flowchart TD
  A["Candidate change: prompt, tool, or model"] --> B["Run agent over eval dataset"]
  B --> C["Score each run"]
  C --> D{"Deterministic checks pass?"}
  D -->|No| E["Fail: wrong tool / bad args / unsafe action"]
  D -->|Yes| F["LLM judge scores reasoning & answer quality"]
  F --> G{"Aggregate score >= release bar & no regressions?"}
  G -->|No| H["Block release, send report"]
  G -->|Yes| I["Promote to production"]

Score what matters: actions, not just words

For financial agents, the most important thing to score is not the prose of the final answer — it is the actions the agent took to get there. Did it call the right tools, in a reasonable order, with correct arguments? Did it avoid calling money-moving tools when the task didn't warrant them? Did it stop and ask when the request was ambiguous? These are largely checkable with deterministic logic against the trace: assert that a reconciliation run never called the transfer tool, that the account it queried matches the account in the task, that no argument was hallucinated.

Layer in outcome scoring on top of the action checks. For tasks with a known correct result — a reconciliation whose true discrepancy you planted — you can assert the agent found exactly that discrepancy. For open-ended tasks where there's no single right answer, you reach for an LLM-as-judge: a separate Claude call given the task, the agent's response, and a rubric, asked to score dimensions like correctness, completeness, and whether the agent stayed within policy. The judge is not infallible, so validate it against human-labeled examples and keep its rubric tight and specific to financial correctness.

Combine deterministic checks with LLM judgment

The strongest eval suites are hybrid. Deterministic checks are cheap, fast, and unambiguous — perfect for the things that have a clear right answer, like "never moves money during a read-only task" or "always cites the source transaction." These run first, and a failure here is an immediate block; there is no debating whether an unsafe action is acceptable. LLM-judge scoring then handles the genuinely subjective dimensions — clarity of an explanation, whether the agent's reasoning was sound, whether it handled an ambiguous request gracefully.

Order matters for cost and signal. Run the cheap deterministic checks first and short-circuit on hard failures, so you only spend judge tokens on runs that passed the non-negotiables. Report scores as distributions, not single averages — a financial agent that's excellent on average but catastrophically wrong on two percent of cases is not shippable, and only a distribution reveals that tail. Track the worst cases as carefully as the mean, because in finance the tail is where the regulatory and reputational risk lives.

Gate releases on the eval, not on vibes

An eval loop only changes behavior if it has teeth. Wire it into your release process so that every candidate change — a new prompt, an added tool, a model swap — runs the full suite automatically and the results gate promotion. Set an explicit release bar: an aggregate score threshold plus a hard rule that no previously passing case may regress. If a change improves the average but breaks a case that used to pass, it does not ship until that regression is understood. This is the discipline that separates teams who ship agents calmly from teams who ship and pray.

Treat model upgrades with the same rigor. When a new Claude version arrives, you do not just assume it's better — you run your eval suite against it and read the diff. Usually the newer model improves most cases, but evals catch the occasional behavior shift that matters for your specific financial workflow, like a change in how aggressively the agent asks for clarification. The eval suite is what lets you adopt model improvements quickly and safely, because you can prove the new model is better on the things you care about rather than hoping.

Close the loop: production feeds the eval

The final piece is making the loop continuous. Production is your richest source of eval cases, so build the pipeline that flows real runs back into the dataset. Sample production runs, flag the ones where the agent struggled or a human had to intervene, and feed those into the suite as new cases. Over time the eval dataset grows to cover the actual distribution of work the agent faces, and the gap between "passes evals" and "works in production" shrinks until they nearly coincide.

This is also where debugging, testing, and operations converge into one practice. A production failure becomes a debugging session, the fix becomes a permanent eval case, and the eval gates the next release. Run that loop consistently and your financial agent gets measurably, monotonically better, with every regression caught before it ships. In a domain where a wrong answer can mean a misposted ledger or a missed fraud flag, that measured confidence is the entire point.

Frequently asked questions

How is an agent eval different from a unit test?

A unit test asserts an exact output and passes or fails deterministically. An agent eval scores behavior against a rubric across many realistic examples and reports a distribution, because agents are non-deterministic and many tasks have several acceptable answers. The goal is measuring relative improvement or regression, not proving perfection.

Should I score the agent's answer or its tool calls?

Both, but in finance the tool calls and actions matter most. Use deterministic checks on the trace to confirm the agent called the right tools with correct arguments and avoided unsafe actions, then add outcome checks and an LLM judge for answer quality. Unsafe actions should be an immediate, non-negotiable failure.

When should I use an LLM-as-judge?

Use a judge for subjective dimensions with no single correct answer — explanation clarity, reasoning soundness, graceful handling of ambiguity. Run cheap deterministic checks first and reserve judge calls for runs that pass them. Validate the judge against human-labeled examples and keep its rubric tightly scoped to financial correctness.

How do evals help with Claude model upgrades?

Run your full eval suite against the new model and read the diff before adopting it. Newer models usually improve most cases, but evals surface behavior shifts that matter for your specific workflow, letting you adopt improvements quickly with proof rather than hope.

Bringing agentic AI to your phone lines

CallSphere runs the same eval discipline behind its voice and chat agents — scoring real conversations and gating releases — so every call is handled to a measured quality bar. See how it's evaluated and shipped at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.