Testing and Evals for Claude Agents in Finance (Verifiable AI Financial Services)

Here is the uncomfortable truth about shipping a financial-services agent: a one-line change to your system prompt can silently regress accuracy on a class of cases you didn't think to check, and you won't find out from any unit test, because the agent still "works." It still answers. It just answers wrong on the 4% of reconciliation cases where the prompt edit nudged it toward the wrong tool. In a domain where wrong means a misstated balance or a missed compliance flag, "it still works" is not a release criterion. You need to measure quality, attach a number to it, and refuse to ship when the number drops.

That measurement discipline is what evals provide. An eval is the agent equivalent of a test suite, but instead of asserting exact equality it scores behavior against a rubric, because there's rarely one correct string — there's a correct outcome. The work is in building a representative case set, choosing graders you trust, and wiring the whole thing into a gate that blocks a release when the score falls below your bar. Done right, it turns prompt engineering from guesswork into something you can iterate on with evidence.

What you're actually measuring

For a financial agent, quality is multidimensional and you should resist collapsing it to one number too early. There's task success: did the reconciliation balance, was the right account flagged. There's tool correctness: did the agent call the right tools with the right arguments, or did it hallucinate an account number that happened to pass validation. There's safety: did it refuse to take an action it shouldn't, gate the money-moving step, decline the out-of-scope request. And there's cost and latency: a correct answer that took forty tool calls and a dollar of tokens is a regression even if the answer is right.

A working definition: an agent eval is a repeatable measurement that runs the agent against a fixed set of representative cases and scores its trajectory and output against explicit criteria, producing a number you can track across versions. The two words that matter are "fixed" and "explicit." Fixed, because a moving case set can't detect regressions. Explicit, because criteria you can't write down are criteria you can't grade consistently.

Building the case set

The case set is where most of the value lives, and it's the part people skimp on. Start by mining production: every failure you debugged, every weird trajectory, every customer complaint becomes a case. A reconciliation that looped, a refund mis-classified, a hallucinated routing number — each one is a regression test the moment you add it, ensuring that specific failure can never silently return. This is how a financial agent's eval set should grow: every bug found is a case added.

Then deliberately seed the edges. Finance is full of them: the leap-year settlement date, the multi-currency transaction, the account with a zero balance, the malformed statement, the injection attempt buried in a memo field. These rarely show up in a happy-path demo but they're exactly where agents fail and where correctness matters most. Aim for coverage across categories rather than volume — fifty cases spanning success, edge, safety, and adversarial buckets beat five hundred near-duplicate happy paths. And keep a clear bucket of cases the agent should refuse or escalate, because measuring that it does the right thing by not acting is as important as measuring that it acts correctly.

flowchart TD
  A["Prompt / tool change"] --> B["Run agent over fixed eval set"]
  B --> C["Collect trajectory + final output"]
  C --> D{"Grader type"}
  D -->|Deterministic| E["Assert tool args, totals, schema"]
  D -->|LLM judge| F["Score vs rubric: success, safety"]
  E --> G["Aggregate score + cost/latency"]
  F --> G
  G --> H{"Score >= release bar & no safety fail?"}
  H -->|Yes| I["Ship"]
  H -->|No| J["Block release, inspect regressions"]

Choosing graders you can trust

Two grader families cover most needs, and you'll use both. Deterministic graders check things with a single right answer: did the reconciliation total match the expected figure, did the agent call get_balance before propose_transfer, were the tool arguments well-formed and within limits. These are cheap, fast, and unambiguous — use them for everything you can express as an assertion, and lean on structured outputs so the agent's results come back in a schema you can check directly rather than parsed out of prose.

For the judgment calls — was the explanation accurate, was the refusal appropriate, did the summary capture the right risk — use an LLM judge: a separate Claude call, in its own context window, that scores the trajectory against an explicit rubric. The discipline here is rubric quality. A vague rubric ("is the answer good?") produces noisy, irreproducible scores. A precise one ("the response must state the exact reconciled balance, cite which transactions were unmatched, and not invent an account number") produces consistent grades. Write rubrics as gradeable criteria, validate your judge against a few human-labeled cases so you trust its calibration, and keep the judge's model and prompt fixed so its scores are comparable across runs.

Gating the release

An eval that nobody acts on is a dashboard, not a gate. The point is to block a regression before it reaches customers, which means the eval runs in CI on every prompt or tool change and the build fails if the score drops below your bar or any safety case fails outright. Treat safety failures as hard blocks regardless of the aggregate: an agent that got 2% more accurate but newly executes an ungated transfer on one adversarial case does not ship. Aggregate quality and per-case safety are different gates, and the safety gate is absolute.

Run the gate at appropriate effort and with the same model you'll deploy, because effort and model choice shift behavior — an eval passing at one effort level tells you little about another. And because agents are non-deterministic, run flaky-sensitive cases a few times and look at the pass rate rather than a single roll; a case that passes four times in five is telling you something different from one that passes every time. The output you want from each CI run is a diff: which cases newly passed, which newly failed, and the cost-latency delta — so a human reviewing the change sees exactly what the edit traded away.

Frequently asked questions

How big does my eval set need to be before it's useful?

Smaller than you think to start, then grow it relentlessly. Even twenty well-chosen cases spanning success, edge, and safety buckets will catch real regressions from day one. The trap is staying small — every production failure you debug should become a case, so the set compounds. A mature financial agent's eval set is mostly cases that were once bugs.

Can I trust an LLM judge to grade financial correctness?

For judgment-laden criteria, yes — once you've calibrated it. Validate the judge against human labels on a sample, give it a precise rubric, and keep its model and prompt fixed. But anything with a single correct answer — a total, a tool argument, a schema — should go to a deterministic grader, not the judge. Use the judge for nuance, deterministic checks for facts.

Should the eval gate run at the same effort as production?

Yes. Effort changes how many tools the agent calls and how it reasons, so an eval at low effort doesn't predict behavior at high effort, and vice versa. Run the gate at the effort and on the model you'll actually deploy, so the number you're gating on reflects what customers will get.

Bring measured quality to your phone lines

Voice agents need the same eval discipline — graded trajectories, regression sets, release gates. CallSphere applies these eval loops to voice and chat agents so quality is measured before it reaches a customer call — see it live at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.