Agentic RAG: AI Agents That Decide When and How to Retrieve Information

What Makes RAG "Agentic"

Standard RAG follows a rigid pipeline: receive a query, embed it, retrieve top-K chunks, pass them to an LLM, and generate an answer. Every question triggers the same retrieval path regardless of whether retrieval is actually needed.

Agentic RAG fundamentally changes this. Instead of a fixed pipeline, an AI agent sits at the center and makes decisions about the retrieval process itself. The agent decides whether to retrieve at all, which sources to query, how to decompose complex questions, and whether the retrieved results are sufficient or need refinement.

This matters because real-world questions are not uniform. A question like "What is Python?" does not need retrieval from your internal knowledge base. A question like "What were Q3 revenue figures for the EMEA region?" requires precise document retrieval. And a question like "Compare our pricing strategy with competitor X across all product lines" requires multi-step planning, multiple retrievals, and synthesis.

The Agentic RAG Architecture

An agentic RAG system has four core capabilities that standard RAG lacks:

flowchart LR
    Q(["User query"])
    REWRITE["Query rewrite<br/>HyDE plus expansion"]
    HYBRID{"Hybrid search"}
    BM25["BM25 keyword<br/>Postgres FTS"]
    DENSE["Dense vector<br/>ANN search"]
    FUSE["Reciprocal rank<br/>fusion"]
    RERANK["Cross encoder<br/>reranker"]
    PACK["Context packing<br/>and dedupe"]
    LLM["LLM generation"]
    OUT(["Cited answer"])
    Q --> REWRITE --> HYBRID
    HYBRID --> BM25 --> FUSE
    HYBRID --> DENSE --> FUSE
    FUSE --> RERANK --> PACK --> LLM --> OUT
    style HYBRID fill:#f59e0b,stroke:#d97706,color:#1f2937
    style RERANK fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style LLM fill:#4f46e5,stroke:#4338ca,color:#fff
    style OUT fill:#059669,stroke:#047857,color:#fff

1. Retrieval decision — The agent evaluates whether external knowledge is needed at all
2. Query planning — Complex questions get decomposed into sub-queries
3. Source routing — Different sub-queries get routed to appropriate data sources
4. Result evaluation — The agent assesses whether retrieved context is sufficient before answering

Building an Agentic RAG System in Python

Here is a practical implementation using LangChain and OpenAI function calling:

from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
from langchain.tools import tool
from langchain.agents import AgentExecutor, create_openai_functions_agent
from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder

# Define retrieval tools for different sources
@tool
def search_product_docs(query: str) -> str:
    """Search internal product documentation for technical details,
    feature descriptions, and usage guides."""
    vectorstore = FAISS.load_local(
        "indexes/product_docs", OpenAIEmbeddings()
    )
    docs = vectorstore.similarity_search(query, k=4)
    return "\n\n".join(d.page_content for d in docs)

@tool
def search_customer_tickets(query: str) -> str:
    """Search customer support tickets for known issues,
    resolutions, and common complaints."""
    vectorstore = FAISS.load_local(
        "indexes/support_tickets", OpenAIEmbeddings()
    )
    docs = vectorstore.similarity_search(query, k=3)
    return "\n\n".join(d.page_content for d in docs)

@tool
def search_financial_reports(query: str) -> str:
    """Search quarterly financial reports for revenue,
    cost, and performance metrics."""
    vectorstore = FAISS.load_local(
        "indexes/financial", OpenAIEmbeddings()
    )
    docs = vectorstore.similarity_search(query, k=3)
    return "\n\n".join(d.page_content for d in docs)

# Build the agent with retrieval tools
llm = ChatOpenAI(model="gpt-4o", temperature=0)

prompt = ChatPromptTemplate.from_messages([
    ("system", """You are a research assistant with access to
    multiple knowledge bases. For each user question:
    1. Decide if retrieval is needed or if you can answer directly
    2. Choose the most relevant source(s) to search
    3. Decompose complex questions into sub-queries
    4. Evaluate if retrieved context fully answers the question
    5. If context is insufficient, search additional sources"""),
    MessagesPlaceholder(variable_name="chat_history"),
    ("human", "{input}"),
    MessagesPlaceholder(variable_name="agent_scratchpad"),
])

tools = [search_product_docs, search_customer_tickets,
         search_financial_reports]
agent = create_openai_functions_agent(llm, tools, prompt)
executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

# The agent decides which tools to use
result = executor.invoke({
    "input": "Why are enterprise customers churning and what "
             "product gaps are driving it?",
    "chat_history": []
})

When given the churn question, the agent autonomously decides to search both customer tickets and financial reports, combines insights from both sources, and synthesizes a coherent answer. A static pipeline could never make this kind of cross-source reasoning decision.

Implementing Query Decomposition

For complex questions, the agent should break them into targeted sub-queries:

from pydantic import BaseModel

class QueryPlan(BaseModel):
    sub_queries: list[str]
    sources: list[str]
    reasoning: str

def plan_retrieval(question: str) -> QueryPlan:
    """Use LLM to decompose a complex question into
    targeted sub-queries with source assignments."""
    response = llm.with_structured_output(QueryPlan).invoke(
        f"""Decompose this question into sub-queries.
        Available sources: product_docs, customer_tickets,
        financial_reports.

        Question: {question}"""
    )
    return response

plan = plan_retrieval(
    "Compare our Q3 churn rate with Q2 and identify "
    "which product issues contributed most"
)
# Returns sub-queries routed to financial + ticket sources

When to Use Agentic RAG

Agentic RAG adds latency and cost compared to standard RAG because the agent must reason about its retrieval strategy. Use it when you have multiple heterogeneous data sources, when questions vary widely in complexity, or when precision matters more than speed. For simple single-source Q&A over uniform documents, standard RAG remains the better choice.

FAQ

How does agentic RAG differ from standard RAG?

Standard RAG always retrieves from a single index using the raw query. Agentic RAG uses an AI agent that decides whether to retrieve, which sources to query, how to decompose questions, and whether results need refinement. The agent adds a reasoning layer on top of the retrieval pipeline.

Does agentic RAG increase latency significantly?

Yes, typically by 1-3 seconds because the agent must make reasoning decisions before and after retrieval. However, for complex multi-source questions, it often produces better answers in fewer total iterations than a naive retrieve-and-retry approach.

Can I use agentic RAG with open-source models?

Absolutely. Any model that supports function calling or tool use can drive an agentic RAG system. Models like Llama 3, Mistral, and Qwen all support the tool-use patterns needed. The key requirement is reliable instruction following for query planning and result evaluation.

---

#AgenticRAG #RAG #AIAgents #QueryPlanning #LangChain #AgenticAI #LearnAI #AIEngineering