GraphRAG: Using Knowledge Graphs to Improve LLM Answers

What GraphRAG is

GraphRAG is a variant of retrieval-augmented generation where the knowledge base is not only a set of text chunks with embeddings, but also a knowledge graph: nodes are entities (people, companies, products, contracts, systems), edges are relations between them ("supplies", "owns", "depends on").

When a question arrives, the system finds relevant entities, walks their connections and passes the connected facts, plus the source text, to the LLM. The model answers from a structured picture of the domain rather than from a handful of similar-looking paragraphs.

How the graph is built

1. Chunking: documents are split into fragments, as in regular RAG.
2. Entity and relation extraction: an LLM (or a classic NER model plus rules) reads each chunk and outputs triples like (Supplier A) -[supplies]-> (Plant B) with a short description and a link to the source chunk.
3. Entity resolution: "Acme LLC", "Acme" and "ACME Ltd." must become one node. This step decides the quality of the whole graph.
4. Storage: the graph goes into a graph database or a relational store with edge tables; chunks and embeddings are usually kept alongside.
5. Optional summaries: some implementations, such as Microsoft's open-source GraphRAG, group the graph into communities of closely related entities and pre-generate a summary for each one.

A minimal extraction prompt output might look like this:

{
  "entities": [
    {"name": "Acme", "type": "company"},
    {"name": "Plant B", "type": "facility"}
  ],
  "relations": [
    {"source": "Acme", "target": "Plant B", "type": "supplies", "chunk_id": "doc12#4"}
  ]
}

How retrieval works

There are two common modes:

• Local search: find entities mentioned in the question, take their neighbours one or two hops away, add the linked source chunks and send everything to the LLM. Good for "Which suppliers of Plant B also work with Plant C?"
• Global search: use community summaries to answer questions about the whole corpus, such as "What are the main risks mentioned across all audit reports?" Vector RAG struggles here because no single chunk contains the answer.

In practice, most production systems are hybrid: vector search finds the entry points, the graph expands context along relations.

When graph retrieval beats plain vector RAG

If your questions are mostly "find the paragraph that says X", vector RAG with good chunking and reranking is the better choice.

The extra cost and complexity

• Indexing cost: every chunk goes through an LLM to extract triples, and summaries add more calls. This is many times more expensive than computing embeddings.
• Updates: new documents mean re-extraction, re-resolving entities and sometimes rebuilding communities. Incremental updates need careful design.
• Extraction errors: the LLM can invent relations or miss them. Errors in the graph propagate into confident wrong answers.
• Entity resolution: duplicates and wrong merges are the most common source of bad results.
• Latency: graph traversal plus larger context can make answers slower.
• More moving parts: an ontology to maintain, a graph store to operate, more things to test.

How to approach it

1. Start with a strong vector RAG baseline and a set of real test questions.
2. Find which questions fail and why. If failures are about connecting facts or summarising the corpus, a graph can help.
3. Define a small ontology: a limited list of entity and relation types relevant to your domain.
4. Build the graph on a subset of documents and compare answers on the same test set.
5. Add human review of extracted entities for critical domains.