Agentic RAG Patterns: When the Agent Decides What to Retrieve

What Changes With Agentic RAG

Classic RAG: a fixed pipeline runs retrieval, then generation. The model has no say in whether to retrieve, what to retrieve, or whether to retrieve again. Agentic RAG: retrieval is one of several tools the agent can call. The agent decides — based on the query and intermediate results — whether to retrieve, what query to use, which corpus to hit, and when to stop.

By 2026 this is the dominant pattern for non-trivial production RAG systems. This piece walks through the patterns that work.

The Five Patterns

flowchart TB
    P1[1. Retrieve-or-skip] --> Use[Skip retrieval if<br/>model already knows]
    P2[2. Multi-source routing] --> Pick[Pick the right corpus]
    P3[3. Query rewriting] --> Better[Rewrite for better recall]
    P4[4. Iterative retrieval] --> Refine[Refine + retrieve again]
    P5[5. Tool-augmented retrieval] --> Mix[Mix vector + SQL + web]

Retrieve-or-Skip

The simplest and most-undervalued pattern. The agent decides whether retrieval is even necessary. Many queries do not need retrieval — small talk, math, code generation, formatting requests. Skipping retrieval saves tokens and avoids irrelevant context polluting the prompt.

Implementation: a short prompt asks the model to classify the query as "needs retrieval" / "answer directly." Any non-trivial RAG system in 2026 has this gate.

Multi-Source Routing

The agent picks which corpus to hit. Production knowledge bases are rarely one homogeneous index — they are a customer FAQ, a product manual, a billing database, a CRM, a web search. The agent classifies the query and routes.

flowchart LR
    Q[Query] --> R[Router Agent]
    R -->|product question| P[(Product KB)]
    R -->|customer question| C[(CRM)]
    R -->|policy question| Pol[(Policy Docs)]
    R -->|external| W[Web Search]

Query Rewriting

The user query is often not the optimal retrieval query. Rewriting expands abbreviations, fixes typos, decomposes multi-part questions, generates hypothetical answers (HyDE).

Iterative Retrieval

The agent retrieves once, examines the results, and decides whether to retrieve again with a different query. This is where CRAG-style refinement lives. Especially valuable on multi-hop questions.

Tool-Augmented Retrieval

Pure-vector RAG is one tool. Real systems mix vector search, SQL queries, knowledge-graph queries, and web search. The agent picks the right combination per query.

A Reference Architecture

flowchart LR
    User --> Agent[Agentic RAG Loop]
    Agent --> D{Decision}
    D -->|skip| Direct[Answer directly]
    D -->|retrieve| Tools[Tool-call retrieval]
    Tools --> Vec[Vector search]
    Tools --> SQL[SQL]
    Tools --> KG[Knowledge graph]
    Tools --> Web[Web search]
    Vec --> Eval[Evaluator]
    SQL --> Eval
    KG --> Eval
    Web --> Eval
    Eval -->|sufficient| Gen[Generate]
    Eval -->|insufficient| Agent
    Gen --> User

The decisions and the loop are what make it agentic. Each step can be a separate small LLM call or fused into the main reasoning model.

Where Agentic RAG Outperforms Classic RAG

The 2026 production data:

• Customer service: 8-15 percent higher resolution rate (the agent knows which subject-matter corpus to hit)
• Internal Q&A: 10-25 percent fewer "I do not know" answers
• Long-form research: 2x quality on multi-hop questions
• Cost: roughly 1.5x classic RAG due to extra LLM calls (retrieve-or-skip pays much of that back)

Common Failure Modes

• Over-retrieval: agent retrieves on every query out of caution; cost balloons. Fix: stricter retrieve-or-skip gate.
• Under-retrieval: agent skips retrieval and confidently hallucinates. Fix: lean toward retrieval on factual questions; calibrate the gate.
• Loop infinity: agent keeps refining and never commits. Fix: hard cap on retrieval rounds.
• Tool selection error: agent hits the wrong corpus. Fix: better router prompts, or a retrieval evaluator that catches mismatches.

Implementing It in 2026

LangGraph, LlamaIndex, and the OpenAI Agents SDK all ship recipes for agentic RAG. The minimal version:

• A "should I retrieve" classifier
• A "which corpus" router
• One or more retrievers as tools
• An evaluator that scores retrieval quality
• A cap on rounds (typically 3)

Most teams ship this in a week. The hard part is not the orchestration; it is the corpora and their evaluators.

Sources

• "Active RAG" Asai et al. — https://arxiv.org/abs/2403.10131
• "Agentic RAG with LangGraph" — https://langchain-ai.github.io/langgraph
• "RAG patterns in 2026" survey — https://arxiv.org
• LlamaIndex agent recipes — https://docs.llamaindex.ai
• "HyDE: hypothetical document embeddings" — https://arxiv.org/abs/2212.10496

Agentic RAG Patterns: When the Agent Decides What to Retrieve — operator perspective

There is a clean theory behind agentic RAG Patterns and there is a messier reality. The theory says agents reason, plan, and act. The reality is that agents stall on ambiguous tool outputs and double-spend tokens unless you put hard limits in place. The teams that ship fastest treat agentic rag patterns as an evals problem first and a modeling problem second. They write the failure cases into the regression set on day one, not after the first incident.

Why this matters for AI voice + chat agents

Agentic AI in a real call center is a different beast than a single-LLM chatbot. Instead of one model answering one prompt, you orchestrate a small team: a router that decides intent, specialists that own a vertical (booking, intake, billing, escalation), and tools that read and write to the same Postgres your CRM trusts. Hand-offs are where most production bugs hide — when Agent A passes context to Agent B, anything that isn't explicit in the message gets lost, and the user feels it as the agent "forgetting." That's why the systems that hold up under load are the ones with typed tool schemas, deterministic state stored outside the conversation, and a hard ceiling on tool calls per session. The cost story is just as important: a multi-agent loop can quietly burn 10x the tokens of a single-LLM design if you let it think out loud at every step. The fix isn't a smarter model, it's smaller agents, shorter prompts, cached system messages, and evals that fail the build when p95 latency or per-session cost regresses. CallSphere runs this pattern across 6 verticals in production, and the rule has held every time: the agent you can debug in five minutes will out-survive the agent that's "smarter" on a benchmark.

FAQs

Q: Why does agentic RAG Patterns need typed tool schemas more than clever prompts?

A: Scaling comes from constraint, not capability. The deployments that hold up keep each agent narrow, cap tool calls per turn, cache the system prompt, and pin a smaller model for routing while reserving the larger model for synthesis. CallSphere's stack — 37 agents · 90+ tools · 115+ DB tables · 6 verticals live — is sized that way on purpose.

Q: How do you keep agentic RAG Patterns fast on real phone and chat traffic?

A: Hard ceilings beat heuristics. A maximum step count, an idempotency key on every tool call, and a fallback to a deterministic script when confidence drops below a threshold are what keep the loop bounded. Evals that simulate noisy inputs catch the rest before they reach a real caller.

Q: Where has CallSphere shipped agentic RAG Patterns for paying customers?

A: It's already in production. Today CallSphere runs this pattern in Real Estate and After-Hours Escalation, alongside the other live verticals (Healthcare, Real Estate, Salon, Sales, After-Hours Escalation, IT Helpdesk). The same orchestrator code path serves voice and chat — the difference is the tool set the router exposes.

See it live

Want to see after-hours escalation agents handle real traffic? Spin up a walkthrough at https://escalation.callsphere.tech or grab 30 minutes on the calendar: https://callsphere.ai/book.