Agentic RAG — When Your Retrieval Pipeline Needs to Think for Itself
The previous post covered two retrieval strategies. One the one side, we have Standard RAG, which finds similar chunks and hands them to the LLM. On the other side, we have Graph RAG, which traverses a knowledge graph, the result can then be handed to an LLM.
Both are single-pass, meaning we have one retrieval step, one generation step, and then it is done.
That is fine for a large class of questions. It is not fine when the question itself determines what needs to be retrieved or even more when the first retrieval reveals that you needed to ask a different question.
Agentic RAG is what happens when you give the LLM control over the retrieval loop.
The Problem with Passive Retrieval
Consider a question like: “Compare the database architectures used in Project Alpha and Project Beta, and identify which approach better aligns with our scalability requirements.”
A Standard RAG system will retrieve the most semantically similar chunks to the full query string. In practice, this will probably find something about Project Alpha and find something about Project Beta. But it will almost certainly not have retrieved the scalability requirements document, because that phrase appeared nowhere near the other two.
The system does not know it retrieved an incomplete picture. It answers with what it has. The type of greedy retrieval is not working in this situation.
Now we could ask, what about the Graph RAG approach. That would probably fail as well, since the scalability document might not mention either Project Alpha or Beta directly and there is no connecting edge either.
An agentic system handles this differently. The LLM looks at the question, decides it needs three separate lookups, performs them in sequence, evaluates what it got, and synthesises from a complete picture.
What “Agentic” Actually Means
An agent, in this context, is an LLM equipped with tools and given permission to decide which tools to call and in what order.
The tools are the retrieval mechanisms, including your vector store, your graph, your web search, your database, whatever is relevant. The LLM reasons about which tool to call, calls it, examines the result, and decides what to do next.
This is a loop, not a pipeline. Especially because the system decides, not you.
Question
|
v
[LLM decides: what do I need?]
|
v
[Tool call: retrieve X]
|
v
[LLM decides: is this enough?]
|
v
[Tool call: retrieve Y] ← only if needed
|
v
[LLM synthesises and answers]
The number of iterations is not fixed. It depends on the question and what the LLM finds.
The Simplest Agentic RAG: ReAct
ReAct (Reason + Act) is the foundational pattern, this basically means that the LLM alternates between thinking out loud and calling a tool.
Gemini’s function-calling capability works cleanly with LangChain’s agent framework. The setup is nearly identical to the standard RAG pipeline from the previous part, swap in ChatGoogleGenerativeAI and the same GoogleGenerativeAIEmbeddings.
In this example we define the Chroma database, set up the connected retriever and then define the knowledge retrieval as a tool. The rest is the setup of the llm, followed by the agent that is defined via create_react_agent() and uses the LLM together with a ReAct workflow. The agent is then attached to an AgentExecutor which can be invoked as usual.
Setting verbose=True is worth doing while you build. You will see the LLM’s reasoning trace, which tool it chose, what it searched for, what it got back, and what it decided to do next. This is how you understand why it fails, not just that it failed.
Something that is of importance here is the definition of prompt, as one can see it comes attached with a safety flag that allows public pull request. In good practice, we should avoid this. We could either define a ReAct workflow manually, as follows
And then change prompt for REACT_PROMPT in create_react_agent().
In general the simple ReAct workflow is a bit outdated. In a more modern setup, we simply replace all agent and result definitions by
The LLM will then decide for itself, whether it has successfully answered the question or invoke another tool
Multi-Tool Agents: Giving the LLM Options
The agentic approach becomes genuinely powerful when you give the LLM multiple retrieval tools. It can choose the right one for each sub-question.
Comparing this to the final of the previous versions, we can see that the change is minimal.
Up to a few small changes to, like not rebuilding the database in this example again, we only add more tools to create_agent(). As examples, we add a web search and a Python function as tools in addition to the existing tool for knowledge retrieval.
The LLM now has three distinct retrieval mechanisms and will call whichever is appropriate for each part of the question. Internal questions go to the vector store. External comparisons go to web search. Team lookups go to the structured tool.
The important part here is that you do not need to hardcode what tool is used when. You let the LLM decide what to do. Of course you can add guidelines in the system_prompt, for example tell the LLM which tools should be prioritised.
The LlamaIndex Alternative: Query Planning Agents
LlamaIndex approaches agentic RAG slightly differently, namely with explicit sub-question decomposition. The SubQuestionQueryEngine breaks a complex question into independent sub-questions, answers each from the appropriate source, and synthesises. A very tiny baby example for this is below.
The engine generates sub-questions like “What is the database architecture of Project Alpha?” and “What is the database architecture of Project Beta?”, executes them in parallel against the appropriate sources, then synthesises the results. Because Settings.llm is set globally, Gemini handles all reasoning and synthesis steps automatically.
This is more structured than ReAct — and less flexible. Useful when you know your retrieval is always a decomposition problem, not when you need the agent to decide whether to retrieve at all. But this can be especially useful if you build an agentic tool that will always combine information from the same set of sources and combine them. Of course it is less useful if your queries can be very varied.
The Library Landscape
Three frameworks dominate agentic RAG implementations.
LangChain + LangGraph
LangChain covers the building blocks, things like tools, agents, chains. LangGraph adds explicit control over the agent loop: you define nodes and edges, making the reasoning flow inspectable and controllable. Use LangGraph when you need to add guardrails, human-in-the-loop steps, or complex conditional routing to your agent.
We will not go into long example here, there are older posts that looked into this.
LlamaIndex
The stronger default choice when the primary product is RAG. Its RouterQueryEngine adds tool selection logic; SubQuestionQueryEngine handles decomposition. This is especially useful if you have a specific repetitive workflow in mind.
As before, there are older posts that go into some details on this.
Haystack
The production-first option. Pipelines are defined declaratively and are easy to serialise, version, and deploy.
Gemini is supported via the GoogleAIGeminiChatGenerator component. Less common in tutorials; more common in enterprise contexts where the pipeline needs to be auditable and reproducible.
LangChain + LangGraph | LlamaIndex | Haystack | |
Agent flexibility | High | Medium | Medium |
RAG primitives | Good | Excellent | Good |
Production tooling | Growing | Growing | Strong |
Best for | Complex, custom agent flows | RAG-first applications | Auditable production pipelines |
Bringing It All Together: Hybrid Agentic RAG
The patterns are not mutually exclusive. A realistic agentic system might use all three retrieval strategies, vector, graph, and structured, as separate tools, letting the LLM decide which to invoke.
We do not really need to give an example here, since we already saw how LangChain can easily add multiple tools to a single agent. In this case we could combine this with the two retrieval versions from the previous post and we already saw an example of a structured function being used as a tool
The LLM then picks the right retrieval mechanism for each sub-question. Semantic search for document content, graph traversal for relationships, and structured lookup for exact records.
This is the composable part. We leave all the tools independent and the reasoning to connect them is completely handled by the agent.
When NOT to Use Agentic RAG
Agents add latency. Each tool call is a round trip. A multi-step agent on a complex question might make four or five LLM calls before it answers, which is expensive and slow compared to a single-pass RAG chain.
They also add unpredictability. A deterministic pipeline always does the same thing. An agent might choose different tool sequences on different runs. That is sometimes a feature, but it can easilty become a problem in production systems.
Use agentic RAG when:
Questions are genuinely multi-step and the steps cannot be predicted in advance.
You have multiple retrieval sources and the right source depends on the question.
Retrieval failure or partial results need to be detected and recovered from.
Stick with standard RAG when:
Questions are well-defined and single-hop.
Latency and cost matter more than answer completeness.
You need deterministic, auditable behaviour.
The Full Picture
We have seen three approaches or rather two approaches and one that can combine the others.
Standard RAG retrieves by similarity. This is fast, simple, and effective for self-contained factual questions. But it has a change to break on multi-document reasoning.
Graph RAG retrieves by structure. It handles relational, multi-hop questions. It is expensive to build, already needing LLM calls for this part. If a connection is missed, it can never retrieve it. It is also unnecessary and even useless, if your domain is not genuinely relational.
Agentic RAG retrieves by reasoning. The LLM decides what to retrieve, evaluates what it got, and retrieves again if needed. This means it has to be given the correct tools, this can include both Standard RAG and Graph RAG or structured information retrieval. This is powerful, but can be slow and is in itself non-deterministic.
These are not competing choices. A well-built agentic system uses standard RAG and graph RAG as tools, different retrieval mechanisms the agent can select from depending on the question. And if the situation does not even warrant a full agent, then a straight forward RAG, either standard or graph might just be good enough.
The right abstraction is the one that is not doing more work than the problem requires.
Comments