Contextual Retrieval Revisited: Anthropic's 2024 Trick in 2026 Practice

TL;DR — Anthropic's 2024 contextual retrieval trick — prepend a 50–100 token explanatory context to each chunk before embedding and BM25 indexing — still wins on most 2026 benchmarks. With Claude prompt caching the indexing cost is ~$1.02 per million document tokens. Combined with a reranker it cuts failed retrievals by ~67% vs vanilla chunking.

The technique

The standard RAG chunking trap: a 200-token chunk reads "Revenue grew 12% this quarter" with no document-level context. The embedding has no idea which company, fiscal quarter, or filing this is. Retrieval grabs noise.

Contextual retrieval fixes this by asking an LLM, for each chunk, "in 50–100 tokens, where does this chunk sit in the parent document?" The output is prepended to the chunk before both embedding and BM25 indexing. Now "Revenue grew 12% this quarter" becomes "From the Q3 2025 ACME Corp 10-Q financial filing, in the discussion of segment performance: Revenue grew 12% this quarter."

flowchart LR
  D[Document] --> C[Chunker]
  C --> CK[Chunk]
  D --> CTX[LLM context generator]
  CK --> CTX
  CTX --> P[Prepended chunk]
  P --> E1[Embed]
  P --> B1[BM25 index]
  E1 --> V[(Vector DB)]
  B1 --> S[(Sparse index)]

How it works

Each chunk is sent with the full parent document to a small LLM (Haiku 4.5 is the recommended fit). The model returns a 50–100 token "where does this fit" string. Both the prepended chunk (for embedding) and the prepended chunk (for BM25) get indexed. At query time, retrieval is identical to vanilla — the cost is paid once at ingest.

Performance from Anthropic's own benchmarks: 35% reduction in failed retrievals with contextual embeddings, 49% with contextual embeddings + contextual BM25, 67% when combined with a Cohere/voyage reranker.

The cost win comes from prompt caching: the parent document is cached for the duration of the indexing pass, so each chunk only pays the marginal cost of the chunk + completion. Net: ~$1.02 per million document tokens with Claude.

CallSphere implementation

CallSphere applies contextual retrieval to every long-form document type: insurance plan booklets, MLS listing PDFs, IT runbooks, vendor contracts. The Healthcare agent retrieves coverage rules with 4–8x better top-1 accuracy when contextual retrieval is on. The OneRoof real-estate agent uses it on listing remarks PDFs where the same phrase ("granite countertops") needs to be tied to the right listing. UrackIT IT helpdesk uses it on multi-section runbooks where step 7 reads "restart the service" with no clue which service.

37 agents · 90+ tools · 115+ tables · 6 verticals · $149/$499/$1499 · 14-day trial · 22% affiliate. Compare retrieval quality across plans on /pricing.

Build steps with code

CTX_PROMPT = """<document>{doc}</document>
<chunk>{chunk}</chunk>
Give a short 50-100 token context that situates this chunk inside the document.
Answer ONLY with the context, nothing else."""

def contextualize(doc, chunk):
    msg = anthropic.messages.create(
        model="claude-haiku-4-5",
        max_tokens=200,
        system=[{"type": "text", "text": "You generate retrieval contexts.", "cache_control": {"type": "ephemeral"}}],
        messages=[{"role": "user", "content": CTX_PROMPT.format(doc=doc, chunk=chunk)}],
    )
    return msg.content[0].text

def index_doc(doc):
    for chunk in chunk_doc(doc, 800):
        ctx = contextualize(doc, chunk)
        prepended = f"{ctx}\n\n{chunk}"
        store_dense(prepended, embed(prepended))
        store_sparse(prepended, bm25_tokenize(prepended))

1. Cache the parent document via Anthropic prompt caching for the whole pass.
2. Always store both the prepended and the original chunk; show originals to the LLM.
3. Re-contextualize on document update; the prepend is parent-version-specific.
4. Use the same embedder for query and chunk; prepend nothing on query side.

Pitfalls

• Prepend leakage: the prepended context can mislead the LLM if shown in the answer. Always strip before final generation.
• Stale context: when the parent doc changes, the prepend is stale. Track version hashes.
• Cost without caching: without prompt caching, the indexing cost is 30x higher.
• Tiny chunks: chunks under 100 tokens get over-shadowed by their own context. Keep chunks 400–1000 tokens.

FAQ

Is this still SOTA in 2026? Yes for most enterprise corpora. ColPali wins on visual; GraphRAG wins on multi-hop global.

Does it stack with hybrid + rerank? Yes — it stacks. Anthropic's own numbers prove it.

Cost at scale? ~$1 per million tokens of documents (not chunks). Cheap.

Can I use a non-Claude model? Yes — same prompt with gpt-4o-mini works. You lose the cache cost edge.

See it on /demo? Yes — switch retrieval mode to "contextual" in the trace view.

Sources

• Contextual Retrieval - Anthropic
• Anthropic's Contextual Retrieval Guide - DataCamp
• Implementing Anthropic's Contextual Retrieval - Instructor
• How To Implement Contextual RAG - Together AI Docs