RAG vs Graph RAG — Two Ways to Give Your LLM a Memory

LLMs are good at reasoning, but they do need to be told what to reason with.

If your application needs to answer questions about documents it has never seen during training, things like a company’s internal knowledge base, a product manual, a legal corpus, you need a way to tell it this information. Fine-tuning is expensive and goes stale, while stuffing the entire document or knowledge base into the prompt smashes any context limits easily.

Retrieval-Augmented Generation is the practical answer. This post is about what RAG actually is, where it quietly breaks, and why Graph RAG exists.

The idea here is not, which one is better, but rather how and when to use it.

ChatGPT has a quite good idea of the difference between the two honestly...

The Visible Part of the Iceberg

At its core, RAG is a two-step idea.

Step one: convert your documents into vectors and store them in a vector database.

Step two: when a question arrives, convert it into the same vector space, find the closest chunks, and hand those chunks to the LLM as context.

In this sense, the LLM never “knows” your documents. Rather, in any situation it reads the relevant excerpts at inference time, like someone who was handed three pages of notes right before being asked a question.

This is simple and surprisingly effective. It is also the source of a specific category of failure that takes a while to notice, but that you can probably easily deduce when thinking about possible examples.

Standard RAG

We look here at a standard exampe of a RAG system in play. Three libraries do most of the heavy lifting here, LangChain for orchestration, a vector store for retrieval, and an embedding model to convert text into vectors.

We will use Gemini throughout, gemini-2.5-flash as the LLM and Google’s text-embedding-001 for embeddings. Both are available via langchain-google-genai. You can easily adapt this to other embeddings and other LLMs using LangChain.

The setup is fairly straight forward, apart from checking the API key, the process works as follows:

1. We load the knowlege base, a simple text file here, use a splitter to create chunks of a given size with given overlap.

2. We use the gemini model to embed this into a vector store, here we use Chroma.

3. We create the retriever and specify it should retrieve the 4 chunks that are most relevant. Up to here, all of this would be done once and only repeated if the knowledge base is changed or updated.

4. Then we define our LLM, our prompt, and finally pipe all of it together. We declare as out workflow to start with the question, retrieve chunks and pass the question through, generate the prompt, send this to the LLM, and finally obtain a string as an output.

5. Here we define our question and invoke our workflow from before.

One thing worth knowing about Gemini embeddings is that document embeddings and query embeddings can use different retrieval modes. In this example we do not call those methods directly. We pass a single GoogleGenerativeAIEmbeddings object into Chroma, and LangChain uses it through its embedding interface: document chunks are embedded when we add them to the vector store, and the user question is embedded when the retriever searches the store.

By default, LangChain’s GoogleGenerativeAIEmbeddings uses RETRIEVAL_DOCUMENT for embed_documents() and RETRIEVAL_QUERY for embed_query(), so the simple version is already doing the right thing for a normal RAG pipeline. If you still need this fine tuning, you can set the retrieval mode for a given object manually.

The Library Options

The above is just a simple example for the package combination. You are by no means locked into LangChain and Chroma. There are plenty of alternatives.

Orchestration frameworks

• LangChain — This is probably the most established and also the most opinionated, at least at the time of this post. It is very good at wiring together even complex pipelines quickly. But the abstraction layer can work against you when you need to debug.

• LlamaIndex — In case the RAG is the main actor of your application, this is a good choice. It features stronger primitives for indexing and retrieval. Worth preferring if RAG is the primary function of your application, not one feature among many.

• Haystack — This is more production-oriented and modular by design. Less commonly seen in tutorials; more commonly seen in production.

Vector stores

• Chroma — local, zero infrastructure, perfect for prototyping.

• FAISS — Meta’s library, CPU-efficient and fast, also local. Good for offline or latency-sensitive applications.

• Qdrant — a proper hosted vector database when you need to scale, filter, and persist.

• Weaviate — adds hybrid search (vector + keyword) out of the box.

• Pinecone — managed, fully hosted, nothing to operate.

The choice of vector store is mostly an infrastructure decision, not a model one. For our experimentation we will use Chroma, but FAISS would be equally easy to put in instead. For production with moderate scale, Qdrant is a good choice, while for large-scale systems with latency constraints, the choice falls on Pinecone or Weaviate.

Where Standard RAG Quietly Breaks

The main drawback, but also the main strength, of Standard RAG is that semantic similarity is the only retrieval signal.

Consider a knowledge base about an organisation, having documents about teams, projects, people, policies. Assume you want to ask: “What is the vacation policy?” Our Standard RAG will most likely work perfectly. There is a high change that the policy document is a dense, self-contained chunk. Similarity retrieval will find it and deliver it as one of the top information chunkgs.

Now ask: “Who should I talk to if I have a problem with the database cluster?”

The answer requires connecting several pieces of information. Let's say the database cluster is managed by Team A. Furthermore Team A’s lead is a Person B. But Person B’s contact is in a different document. The relationship between the cluster, the team, and the person is never stated in a single chunk.

Standard RAG retrieves fragments. The LLM has to infer the connections, if they are even present in the retrieved chunks, which is not guaranteed. For example in our hypothetical above, the retrieval might return all documents about the cluster, including which person to contact, but it will not retrieve the document with Person B's contact details in the first try.

This is the structural gap that Graph RAG addresses.

The Idea Behind Graph RAG

The basic premise here is: Documents do not exist in isolation. They describe entities and the relationships between them. A knowledge graph makes those relationships explicit.

Graph RAG replaces the flat vector index with a knowledge graph. Entities, that could be people, products, concepts, policies, become nodes and relationships become edges. Retrieval traverses the graph rather than computing similarity scores.

The result: queries that require multi-hop reasoning become answerable in a principled way, not by luck of chunk proximity. In our hypothetical example from above, a part of our knowledge graph would contain the following edges

[ Database Cluster ] --managed_by--> [ Team A ]
[ Team A ] --has_lead--> [ Person B ]
[ Person B ] --reachable_via --> [ Slack: @personb ]

A graph traversal answers the multi-hop question directly. A similarity search hopes the answer appeared somewhere in a single chunk.

Graph RAG in Practice

Standard RAG is easy to demonstrate because it is just a vector store and a retriever. Graph RAG is different in this regard, since there is an indexing step where the system has to read your documents, extract entities, infer relationships, build graph structures, and prepare them for query-time retrieval.

That means the first working example looks less like a normal Python script and more like a small project setup.

We will use Microsoft’s graphrag package and Gemini as both the generation model and the embedding model.

The example will use three tiny fake documents:

1. A database cluster document saying that a database cluster, called AtlasDB, is managed by Team A.

2. A team document saying that Person B is the Team Leader of Team A. Plus any other fake roles you want to appoint in the team.

3. A contact document giving contact details for everyone in Team A.

The point is that the answer is not stated in one place. The question:

Who should I contact about the database cluster?

requires connecting several facts:

AtlasDB Cluster
  -> managed by Team A
  -> Team A is led by Person B
  -> Person B can be contacted by ... whatever you added

That is exactly the kind of query Graph RAG is meant to handle.

Step 1: Initialise the graphrag project

After adding graphrag, there are a few steps to set up. From inside the project folder, run:

graphrag init --root .

This creates the basic project structure:

my_project/
  input/
  .env
  settings.yaml

The important parts here are:

input/

where your source documents go,

.env

where your API key goes, and finally

settings.yaml

where the LLM and embedding models are configured.

Step 2: Add the input documents

Put your source text files into the input/ folder. As mentioned above we use three small documents

my_project/
  input/
    01_database_cluster.txt
    02_team_a_people.txt
    03_team_a_contacts.txt

that contain the informations needed. If you want, you can easily have these generated by ChatGPT or any LLM of your choice, just describe the necessary information. The point here is that a normal vector search may or may not retrieve all three of these pieces together. Graph RAG tries to make the relationship explicit during indexing.

Step 3: Add the Gemini API key

Open the .env file and add your Gemini key:

GEMINI_API_KEY=your-google-ai-studio-key

Do not paste your API key directly into the Python code or the YAML file. Keep it in .env.

Also be careful with the variable name. If you leave the default OpenAI-oriented configuration in place, graphrag will try to send your Gemini key to OpenAI. The resulting error is confusing but predictable:

Incorrect API key provided

That does not mean your Gemini key is necessarily wrong. It usually means the model provider is still configured as OpenAI.

This even happens when putting in gemini for the completion and embedding models when graphrag is added, since the setting.yaml file will still link to OpenAI as a provider.

Step 4: Configure Gemini in settings.yaml

The exact file generated by graphrag init may contain more settings than we need to discuss here. The key part is the model configuration. For Gemini, the relevant sections should look like this:

completion_models:
  default_completion_model:
    model_provider: gemini
    model: gemini-2.5-flash-lite
    auth_method: api_key
    api_key: ${GEMINI_API_KEY}

embedding_models:
  default_embedding_model:
    model_provider: gemini
    model: gemini-embedding-001
    auth_method: api_key
    api_key: ${GEMINI_API_KEY}

The important line is this:

model_provider: gemini

If that still says:

model_provider: openai

then graphrag will treat your key as an OpenAI key. This part may also look considerably different if you use an older version than available at the time of this post.

Step 5: Build the graph index

Now we finally run the indexer:

graphrag index --root .

This is the expensive step where graphrag reads the documents in input/, sends them through an LLM-powered extraction pipeline, and writes the generated index into the project output directory.

Conceptually, this step tries to discover things like:

AtlasDB Cluster -> managed_by -> Team A
Team A -> has_leader -> Person B
Person B -> reachable_via -> @person-b
Person B -> has_email -> person.b@example.internal

The exact graph is not hand-written by us. It is inferred from the source documents.

That is powerful, but also important to understand. Graph RAG depends on extraction quality. If the indexing model misses an entity or relationship, query-time retrieval cannot use that missing edge later.

This is one reason Graph RAG is more expensive and more fragile than Standard RAG. It does more work before you ever ask a question. But if it misses a step, you cannot fix it by letting the receiver return more chunks or something similar, you have to re-index everything.

Step 6: Ask a local GraphRAG query

Once indexing has completed, ask a question with local search:

graphrag query \
  --root . \
  --method local \
  "Who should I contact about the database cluster?"

Local search is the right mode for this example because the question is about specific entities:

database cluster
Team A
Team Leader
Person B

A good answer should say something like:

You should contact Person B, the Team Leader of Team A. Team A manages AtlasDB Cluster, and Person B is listed as the preferred contact for production database ownership questions. Person B can be reached at person.b@example.internal or on Slack at @person-b.

The interesting part is not the answer itself, it is that no single document had to say the full sentence:

Contact Person B about AtlasDB Cluster.

The answer is assembled from relationships across documents. This is the Graph RAG idea in miniature.

What This Example Shows

This example is deliberately tiny. A normal RAG system can answer questions when the answer is sitting inside one or two relevant chunks. Graph RAG becomes more interesting when the answer depends on relationships:

system -> team -> team leader -> contact method

We did not show any example here, but there is a Python API for graphrag as well.

"Light-weight" Alternative

For lighter-weight graph retrieval, LlamaIndex has a KnowledgeGraphIndex that works with Neo4j or local graph stores. For this, one first needs to set up a Neo4j database. It will depend on your system and choice of either using a docker container or the desktop application. But the code afterwords would look as follows:

After setting up the LLM and embedding, we load the documents and initialize the graph storage in the Neo4j database. Here "my-secret-password" is anything you use for the database password and the url assumes that the standard connection port. Then we build the index. All of this should of course only be done once in a production model. Then we query the engine with our request. To not get any errors we finish by closing the database.

This is also a good opportunity to show a hybrid retrieval model, where a graph and a vector retriever are used together. We have to add two imports, configure two different types of retrievers, and then add them as sub-retrievers to the query engine.

The Real Difference

Essentially, standard RAG retrieves by proximity, while graph RAG retrieves by structure.

Neither is universally better. The right choice depends on what your documents actually contain.

	Standard RAG	Graph RAG
Best for	Self-contained factual documents	Relational, interconnected knowledge
Retrieval signal	Semantic similarity	Graph traversal
Setup cost	Low	High (graph extraction is expensive)
Query types	Lookup, summarisation	Multi-hop, relationship, causal
Failure mode	Missing cross-document connections	Incomplete graph extraction
Good starting point	LangChain + Chroma + Gemini	graphrag or LlamaIndex + Neo4j + Gemini

When NOT to Use Graph RAG

Graph extraction relies on an LLM to identify entities and relationships. That process is imperfect, expensive, and slow.

If your documents are mostly independent, whether this is articles, support tickets, product descriptions, or something similar, Graph RAG adds complexity without adding value. The graph will be sparse and Standard RAG will outperform it for less cost.

Graph RAG earns its keep when your knowledge has genuine relational structure: org charts, codebases, regulatory frameworks with cross-references, ontologies. If you cannot draw a meaningful node-edge diagram of your domain, you probably do not need a graph.

What Comes Next

Both approaches so far are passive. They retrieve and then answer, they do not acutally decide what to do at any given step.

Sometimes retrieval is not a single step. Sometimes the LLM needs to decide what to retrieve, look at the result, decide what else to retrieve, and assemble the answer iteratively.

That is Agentic RAG and that is what the next post is about.