How to Measure Citation Grounding in Claude Systems

"It cites its sources now" is not a measurement. It's a vibe. Plenty of systems display tidy citations on every line and still attach those citations to claims the sources never made. If you can't put a number on whether your Claude answers are actually grounded, you don't know whether the citations are protecting your users or just decorating your hallucinations. This post is about the specific metrics and signals that turn "it feels grounded" into "we measure 96% claim support and we'd catch a regression in a day."

Key takeaways

• The headline metric is claim-support rate: the fraction of factual sentences whose citation genuinely backs them.
• Track abstention correctness too — a system that never says "I don't know" is gaming you.
• Separate retrieval quality from attribution quality; they fail for different reasons and need different fixes.
• Build a labeled eval set with known answers and known unanswerable questions, then run it on every change.
• Watch leading signals in production — citation density, abstention rate, source diversity — to catch drift before users do.

What does "working" even mean here?

Faithfulness, in grounded generation, is the degree to which every claim in an answer is supported by the cited source material and nothing is asserted beyond it. That definition gives you two things to measure separately: are the claims supported (faithfulness), and did the system answer the questions it should while declining the ones it can't (coverage and abstention). A system can be perfectly faithful and useless if it abstains on everything, or highly responsive and dangerous if it never abstains.

Which metrics actually matter?

Four numbers carry most of the signal. Claim-support rate is the share of factual sentences whose cited span truly supports them — your faithfulness headline. Citation precision is, of the citations present, how many point at a span that supports the claim. Answerable coverage is the share of answerable questions the system actually answers rather than over-abstaining. And correct-abstention rate is how often it declines exactly the questions the corpus can't support.

flowchart TD
  A["Labeled eval set"] --> B{"Question answerable
from corpus?"}
  B -->|Yes| C["Expect cited answer"]
  B -->|No| D["Expect abstention"]
  C --> E["Run Claude grounded"]
  D --> E
  E --> F{"Judge: claims supported?"}
  F -->|Supported| G["Score claim-support rate"]
  F -->|Unsupported| H["Log as faithfulness miss"]
  G --> I["Dashboard: trend over releases"]
  H --> I

The crucial design choice in this loop is including unanswerable questions in the eval set. Without them, you only measure how well the system answers — never how well it knows when to stop. A grounding system that scores 99% on answerable questions but answers 80% of unanswerable ones is a liability, and only a mixed eval set reveals it.

How do you score faithfulness automatically?

At scale you can't hand-grade every answer, so use Claude as a judge with a strict, narrow rubric. The judge sees only a claim and its cited span and rules on support — nothing else, so it can't rationalize. Here's a runnable scoring stub:

def score_answer(claims):  # claims: list of (sentence, cited_span)
    results = []
    for sentence, span in claims:
        verdict = judge(  # one Claude call, strict rubric
            f"CLAIM: {sentence}\nSPAN: {span}\n"
            "Reply SUPPORTED, PARTIAL, or NOT_SUPPORTED. "
            "NOT_SUPPORTED if the span does not directly state it.")
        results.append(verdict)
    supported = sum(v == "SUPPORTED" for v in results)
    return supported / len(results)  # claim-support rate

Run this across your eval set on every prompt or retrieval change, and you get a single trend line that tells you whether a tweak helped or quietly regressed faithfulness. Validate the judge itself against a few hundred human labels first, then trust it for the bulk.

Common pitfalls in measurement

• Measuring only answered questions. If your eval set has no unanswerable questions, you can't detect over-confident answering. Always mix in known unknowns.
• Conflating retrieval and attribution. A low score might mean the right source wasn't retrieved (fix retrieval) or that it was retrieved but mis-cited (fix the prompt). Measure each separately.
• Trusting an unvalidated judge. An LLM judge with a loose rubric grades generously. Calibrate it against human labels before you rely on its numbers.
• Vanity citation density. More citations per answer looks rigorous but can mean the model is sprinkling sources to seem grounded. Measure support, not count.
• No production signals. Offline evals catch known cases; production drift comes from new questions and changed sources. Monitor live abstention and support rates too.

Stand up measurement in five steps

1. Build a labeled eval set of 150–300 questions, including a deliberate slice of unanswerable ones.
2. Define claim-support rate as your headline metric and pick a target (many teams aim for 95%+).
3. Calibrate a Claude judge against human labels, then use it to score claim-by-claim support.
4. Run the eval on every retrieval or prompt change and chart the trend by release.
5. Add production signals — abstention rate, citation precision sampling, source diversity — and alert on drift.

Metric to signal to fix

Frequently asked questions

What's a good claim-support rate?

Many production teams target 95%+ on their eval set and treat any release that drops below their baseline as a blocker. The right bar depends on stakes — regulated answers demand higher.

Can I trust an LLM as the judge?

Only after calibrating it against human labels on a sample. A validated judge with a strict rubric correlates well with human grading; an uncalibrated one inflates scores.

Offline evals or production monitoring?

Both. Offline evals gate releases; production signals catch drift from new questions and changing sources that your eval set never saw.

Measurable, grounded AI on the phone

The same metrics that prove a text system is grounded prove a voice agent is trustworthy. CallSphere instruments its voice and chat agents for faithfulness and correct hand-offs so you can see, not guess, that every answered call stays grounded. See it at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.