RAG Failure Mode Catalog: Why Pipelines Don't Find the Right Doc

Why a Catalog

Production RAG systems fail in repeating ways. Knowing the catalog lets you diagnose quickly. Most "the AI gave a wrong answer" reports trace back to one of twelve failure modes documented across 2025-2026 production systems.

This piece is the working catalog.

The Twelve

flowchart TB
 F[Failure modes] --> F1[1. Wrong chunk]
 F --> F2[2. Lost in middle]
 F --> F3[3. Stale corpus]
 F --> F4[4. Embedding model mismatch]
 F --> F5[5. Chunk too small]
 F --> F6[6. Chunk too large]
 F --> F7[7. Vocabulary gap]
 F --> F8[8. Reranker confused]
 F --> F9[9. Cross-tenant leak]
 F --> F10[10. Coverage gap]
 F --> F11[11. Conflicting docs]
 F --> F12[12. PII / privacy leak]

1. Wrong Chunk

The retriever returned a relevant-looking but actually wrong chunk. Common with broad keywords.

Fix: stronger reranker; query rewriting; hybrid retrieval.

2. Lost in Middle

The right chunk was retrieved but the LLM ignored it because of position in the prompt.

Fix: rerank to put best chunks first; use shorter context windows; structured separators.

3. Stale Corpus

The corpus has not been re-indexed since a relevant document was added or updated.

Fix: streaming index updates; corpus version tracking; freshness metrics.

4. Embedding Model Mismatch

Queries embedded with one model, corpus with another. Distance computations are nonsense.

Fix: re-embed corpus when embedding model changes; tag embeddings with model version.

5. Chunk Too Small

Chunks are 100 tokens; the relevant context is in the surrounding 500 tokens. Retrieval gets the chunk; the model lacks context to use it.

Fix: larger chunk sizes; chunk overlap; expanded context retrieval.

6. Chunk Too Large

Chunks are 2000 tokens; relevant facts dilute among irrelevant content. Embedding does not represent any single concept well.

Fix: smaller chunks; semantic chunking; multi-granularity indexing.

7. Vocabulary Gap

Domain terminology not represented in the embedding model. Codes, abbreviations, technical terms miss.

Fix: domain-tuned embeddings; hybrid retrieval (BM25 catches exact matches); vocabulary expansion.

8. Reranker Confused

Cross-encoder reranker shifts the wrong chunk to the top.

Fix: use a stronger or domain-tuned reranker; combine reranker with RRF fallback; validate rerank improvements on your data.

9. Cross-Tenant Leak

Documents from tenant A retrieved for tenant B's query.

Fix: per-tenant indexes; per-tenant filters baked into every query; audit log of retrievals.

10. Coverage Gap

The right document is not in the corpus at all.

Fix: corpus auditing; coverage testing on known questions; expansion of source corpora.

11. Conflicting Docs

Two retrieved documents contradict each other; the LLM confidently picks one.

Fix: explicit conflict-resolution prompts ("if sources conflict, note the conflict"); date-aware ranking; provenance tracking.

12. PII / Privacy Leak

Sensitive data appears in retrieved chunks where it should not.

Fix: PII redaction at index time; access-control filtering at retrieval time; redaction at generation time.

Diagnosis Workflow

flowchart LR
 Bad[Bad answer reported] --> Trace[Pull trace]
 Trace --> Check[Check retrieved chunks]
 Check --> Match{Right chunks?}
 Match -->|No| RetFail[Retrieval failure: 1, 3, 4, 5, 6, 7, 8, 10]
 Match -->|Yes| GenFail[Generation failure: 2, 11, 12]

Was the retrieval wrong, or did the model fail to use right retrieval correctly? Different failures, different fixes.

Test Cases for Each

A 2026 RAG eval suite should include tests targeting each failure mode:

• Wrong chunk: ambiguous queries
• Lost in middle: long contexts with answer late
• Stale corpus: queries about recent updates
• Cross-tenant: multi-tenant test data
• Coverage gap: known-not-in-corpus queries

If you do not test for them, you discover them in production.

Sources

• "RAG failure modes" Hamel Husain — https://hamel.dev
• "Analyzing RAG failures" research — https://arxiv.org
• LangSmith eval patterns — https://docs.smith.langchain.com
• "RAG production debugging" Anthropic — https://www.anthropic.com/engineering
• "Lost in the middle" Liu et al. — https://arxiv.org/abs/2307.03172

RAG Failure Mode Catalog: Why Pipelines Don't Find the Right Doc: production view

RAG Failure Mode Catalog: Why Pipelines Don't Find the Right Doc usually starts as an architecture diagram, then collides with reality the first week of pilot. You discover that vector store choice (ChromaDB vs. Postgres pgvector vs. managed) is not really a vector store choice — it's a latency, freshness, and ops choice. Picking wrong forces a re-platform six months in, exactly when you have customers depending on it.

Broader technology framing

The protocol layer determines what's possible: WebRTC for browser-side widgets, SIP trunks (Twilio, Telnyx) for PSTN voice, WebSockets for the Realtime API streaming session. Each has its own jitter buffer, its own ICE/STUN dance, and its own failure modes when a customer's corporate firewall is hostile.

Front-end is Next.js 15 + React 19 for the marketing surface and the in-app dashboards, with server components used heavily for the SEO-critical pages. Backend splits across FastAPI for the AI worker, NestJS + Prisma for the customer-facing API, and a thin Go gateway that does auth, rate limiting, and routing — letting each service scale on its own characteristics.

Datastores: Postgres as the source of truth (per-vertical schemas like healthcare_voice, realestate_voice), ChromaDB for RAG over support docs, Redis for ephemeral session state. Postgres RLS enforces tenant isolation at the row level so a misconfigured query can't leak across customers.

FAQ

Why does rag failure mode catalog: why pipelines don't find the right doc matter for revenue, not just engineering? The healthcare stack is a concrete example: FastAPI + OpenAI Realtime API + NestJS + Prisma + Postgres healthcare_voice schema + Twilio voice + AWS SES + JWT auth, all HIPAA aligned. For a topic like "RAG Failure Mode Catalog: Why Pipelines Don't Find the Right Doc", that means you're not starting from scratch — you're configuring an agent template that's already been hardened across thousands of conversations.

What are the most common mistakes teams make on day one? Day one is integration mapping (scheduler, CRM, messaging) and prompt tuning against your top 20 real call transcripts. Day two through five is shadow-mode running, where the agent transcribes and recommends but a human still answers, so you can compare side-by-side. Go-live is the moment your eval pass-rate clears your internal bar.

How does CallSphere's stack handle this differently than a generic chatbot? The honest answer: it scales until your tool catalog gets stale. The agent is only as good as the integrations it can actually call, so the operational discipline is keeping schemas, webhooks, and fallback paths green. The platform handles the rest — observability, retries, multi-region routing — without your team owning the GPU layer.

Talk to us

Explore a live demo and compare current plans to find the right fit for your business.