Gen AI

Graph RAG

Instead of searching independent text chunks, Graph RAG extracts entities and relationships into a knowledge graph, allowing the LLM to traverse connections to answer highly complex, multi-hop questions.

Vector search retrieves isolated chunks. Graph RAG connects those chunks via a knowledge graph, allowing the system to traverse relationships across the entire dataset.

Why Does This Exist?

Standard Retrieval-Augmented Generation (RAG) relies on vector search. Vector search is phenomenal at finding a specific needle in a haystack if you know exactly what the needle looks like. If you ask, "What were Apple's Q3 revenues?", vector search finds the exact paragraph mentioning Apple's Q3 revenue instantly.

However, standard RAG fails completely on global, multi-hop, or relational queries. If you ask: "How are the political tensions in Taiwan affecting our company's supply chain?"

A standard vector database will retrieve:

A document about Taiwan politics.
A document about your company's supply chain.

But it won't retrieve the crucial documents connecting them, because those documents might not mention "Taiwan" or "your company" at all. They might mention "TSMC" (a Taiwanese chipmaker) and "Vendor X" (your supplier who buys from TSMC).

Graph RAG solves this by converting your unstructured text documents into a structured Knowledge Graph during ingestion. When a user asks a complex question, the system traverses the graph, following the explicit relationships between entities (Taiwan $\rightarrow$ TSMC $\rightarrow$ Vendor X $\rightarrow$ Your Company) to build a comprehensive answer.

Think of It Like This

The detective's string board

Standard RAG is a detective staring at a massive filing cabinet. If they need information about a suspect named "John," they pull every file with the word "John" in it, read them in a pile, and try to make sense of it.

Graph RAG is a detective's string board. They read a file, pin a photo of "John" to the board, draw a red string to a photo of "Sarah" (his sister), and draw another string to "The Bank" (where Sarah works). When someone asks, "How is John connected to the bank robbery?", the detective doesn't just read John's file; they follow the red string on the board.

How It Actually Works

Graph RAG fundamentally alters both the ingestion and retrieval phases of the pipeline.

1. Ingestion: Entity and Relationship Extraction

You cannot just dump text into a graph database (like Neo4j or Memgraph). You must structure it first. During the ingestion pipeline, an LLM is prompted to act as an information extractor. It reads a chunk of text and identifies:

Entities (Nodes): People, places, organizations, concepts. (e.g., "Apple", "Tim Cook").
Relationships (Edges): How the entities interact. (e.g., "CEO_OF").

The LLM outputs a triplet: (Tim Cook) -[CEO_OF]-> (Apple). These triplets are loaded into the graph database.

2. Community Detection (Microsoft GraphRAG approach)

Advanced systems (like Microsoft's open-source GraphRAG) go a step further. Once the massive graph is built, they use graph clustering algorithms (like Leiden) to group closely connected nodes into "Communities." The system then uses an LLM to generate a summary of each community. It creates a hierarchy: summaries of the highest-level themes, breaking down into summaries of sub-themes, all the way down to individual node relationships.

3. Retrieval and Generation

When a user asks a global question like, "What are the primary risk factors mentioned across all our corporate reports?":

Local Search: The system can find a specific entity (e.g., "Risk Factor A") and traverse its immediate edges to find related entities.
Global Search: The system queries the pre-computed community summaries. Instead of trying to retrieve 10,000 individual sentences, it retrieves the high-level summaries generated during ingestion, allowing the LLM to answer questions that require "understanding the whole dataset."

Show Me the Code

Extracting entities and relationships is the hardest part of building a Graph RAG system. Here is a conceptual example of how to prompt an LLM to generate graph triplets.

1import openai2
3def extract_graph_triplets(text_chunk):4    system_prompt = """5    You are an expert data extraction algorithm. Your job is to extract entities and 6    relationships from the provided text to build a knowledge graph.7    8    Output a list of triplets in the format: (Entity1) -[RELATIONSHIP]-> (Entity2)9    Do not output any other text.10    """11    12    response = openai.chat.completions.create(13        model="gpt-4o",14        messages=[15            {"role": "system", "content": system_prompt},16            {"role": "user", "content": text_chunk}17        ],18        temperature=0.019    )20    21    return response.choices[0].message.content22
23# --- Example ---24document = """25Acme Corp recently acquired Globex Inc for $4 billion. Jane Doe, the CEO of Acme Corp, 26stated that the acquisition will help them expand into the European market. Globex Inc 27is headquartered in Berlin.28"""29
30triplets = extract_graph_triplets(document)31print(triplets)32
33# -> (Acme Corp) -[ACQUIRED]-> (Globex Inc)34# -> (Acme Corp) -[ACQUISITION_PRICE]-> ($4 billion)35# -> (Jane Doe) -[CEO_OF]-> (Acme Corp)36# -> (Jane Doe) -[STATED]-> (expand into European market)37# -> (Globex Inc) -[HEADQUARTERED_IN]-> (Berlin)38
39# These triplets would then be inserted into Neo4j using Cypher queries.

Watch Out For

Extreme Ingestion Costs

Standard RAG ingestion is incredibly cheap—you just run text through a fast, cheap embedding model. Graph RAG ingestion requires running every single chunk of text through an expensive, reasoning LLM (like GPT-4) to extract the entities and relationships. If you have 10,000 documents, building the initial knowledge graph can easily cost thousands of dollars in API credits and take days to process.

Entity Resolution (The duplication nightmare)

If Document A mentions "Apple," Document B mentions "Apple Inc," and Document C mentions "Apple Corporation," a naive LLM extractor will create three separate nodes in your graph. Your relationships will be fragmented, and traversals will fail. Building a Graph RAG system requires a robust "entity resolution" step to merge identical entities under a single canonical name before loading them into the database.

The Quick Version

Standard vector search fails at answering broad, holistic questions or questions requiring multi-hop logical deductions.
Graph RAG converts unstructured text into a Knowledge Graph (nodes and edges) during ingestion.
This requires an LLM to read the text and extract explicit relationships (e.g., (Person) -[WORKS_AT]-> (Company)).
During retrieval, the system traverses these relationships, allowing it to connect disparate pieces of information across the entire dataset.
It provides vastly superior answers for complex queries, but at a massively increased ingestion cost.