When Does Graph RAG Actually Add Value? A Hands-On Experiment
When Does Graph RAG Actually Add Value? A Hands-On Experiment
图谱 RAG 究竟何时能带来价值？一次实战实验
Over the last year, I’ve lost count of the number of conversations I’ve had about AI agents, RAG, semantic layers, and knowledge graphs. Whether you’re reading analyst reports, vendor architectures, or technical blogs, knowledge graphs increasingly show up as a key ingredient for helping AI systems understand not just information, but the relationships between pieces of information.
在过去的一年里，我已经数不清进行过多少次关于 AI 智能体、RAG（检索增强生成）、语义层和知识图谱的对话了。无论你是在阅读分析师报告、厂商架构文档还是技术博客，知识图谱正日益成为一个关键要素，帮助 AI 系统不仅理解信息本身，还能理解信息片段之间的关联。
I had no reason to doubt that. But I wanted to understand it a bit more concretely. Specifically, I wanted to know whether adding a knowledge graph would materially improve the quality of answers for a retrieval use case, and whether the additional engineering effort was justified when compared with more traditional RAG approaches (or simply dumping the source material directly into the context window of a frontier model).
我没有理由怀疑这一点，但我希望能更具体地理解它。具体来说，我想知道在检索用例中，引入知识图谱是否能实质性地提高回答质量，以及与传统的 RAG 方法（或者仅仅将源材料直接丢入前沿模型的上下文窗口）相比，额外的工程投入是否值得。
So, I built four different approaches, ran them against the same documents and questions, and compared the results. The outcome was clear: the frontier model produced the strongest answers overall. At first glance, that might sound like a disappointing result for Graph RAG. It wasn’t. In fact, the more I looked at the outputs, the more interesting the trade-offs became. Graph RAG consistently outperformed traditional RAG on some types of questions, struggled on others, and highlighted something that is often missing from the discussion: the value of a graph depends heavily on the problem you’re trying to solve.
因此，我构建了四种不同的方法，用相同的文档和问题对它们进行了测试，并比较了结果。结论很明确：前沿模型总体上给出了最强的回答。乍一看，这对于图谱 RAG 来说似乎是一个令人失望的结果。但事实并非如此。事实上，我越深入研究输出结果，就越发现其中的权衡非常有趣。图谱 RAG 在某些类型的问题上始终优于传统 RAG，但在其他问题上表现吃力，这凸显了一个在讨论中经常被忽略的事实：图谱的价值在很大程度上取决于你试图解决的问题。
This article isn’t really about which approach won. It’s about understanding when the additional complexity of Graph RAG is justified, and when it probably isn’t.
本文并非旨在讨论哪种方法胜出，而是为了探讨何时引入图谱 RAG 的额外复杂性是合理的，以及何时它可能是不必要的。
The Experiment
实验过程
Before diving into the results, it’s worth explaining what I was trying to achieve. This wasn’t intended to be a rigorous benchmark. The dataset was deliberately small, consisting of two documents (I used two recently published articles by Anthropic on the topic of AI Security) and a set of evaluation questions. My goal wasn’t to identify a universally superior architecture. I wanted to understand how different retrieval approaches behave when given access to the same information, and more specifically, whether the additional complexity of a knowledge graph translates into better answers.
在深入探讨结果之前，有必要解释一下我的目标。这并非旨在成为一个严谨的基准测试。数据集被刻意保持得很小，仅包含两份文档（我使用了 Anthropic 最近发表的关于 AI 安全的两篇文章）和一组评估问题。我的目标不是确定一种普遍优越的架构，而是想了解当面对相同信息时，不同的检索方法会有何表现，更具体地说，知识图谱带来的额外复杂性是否真的能转化为更好的回答。
To keep the comparison as fair as possible, I used the same core components across all retrieval-based systems. I used Microsoft’s Phi-4 language model (14 billion parameters) for the answer-generation model and ran it locally on my Thinkpad through Ollama. I used the all-MiniLM-L6-v2 embedding model for powering semantic search. For the knowledge graph and graph-based retrievals, I used Neo4j to store and query entities and relationships extracted from the source material.
为了尽可能保证比较的公平性，我在所有基于检索的系统中使用了相同的核心组件。我使用微软的 Phi-4 语言模型（140 亿参数）作为答案生成模型，并通过 Ollama 在我的 Thinkpad 上本地运行。我使用 all-MiniLM-L6-v2 嵌入模型来支持语义搜索。对于知识图谱和基于图的检索，我使用 Neo4j 来存储和查询从源材料中提取的实体和关系。
System A — Plain RAG
系统 A — 基础 RAG
This is the architecture most people think of when they hear the term RAG. Documents are broken into chunks and converted into vector embeddings. When a user asks a question, the system retrieves the most relevant chunks and provides them to the language model as context. The model then generates an answer using only the retrieved passages. This served as the baseline against which all other approaches were compared.
这是大多数人在听到 RAG 这个术语时想到的架构。文档被拆分成块并转换为向量嵌入。当用户提出问题时，系统检索最相关的块，并将它们作为上下文提供给语言模型。模型随后仅使用检索到的段落生成答案。这作为基准，用于与其他所有方法进行比较。
Under the Hood: I used LangChain’s RecursiveCharacterTextSplitter to split the documents into 500-character chunks with a 50-character overlap. Each chunk was then converted into a 384-dimensional embedding using the all-MiniLM-L6-v2 model and stored in memory. At query time, the question was embedded using the same model and cosine similarity was calculated against all chunk vectors using NumPy. The five most relevant chunks were retrieved and supplied to Phi-4 (running locally via Ollama) along with the original question as context.
技术细节：我使用 LangChain 的 RecursiveCharacterTextSplitter 将文档拆分为 500 个字符的块，并保留 50 个字符的重叠。每个块随后使用 all-MiniLM-L6-v2 模型转换为 384 维的嵌入并存储在内存中。在查询时，问题使用相同的模型进行嵌入，并使用 NumPy 计算与所有块向量之间的余弦相似度。最相关的五个块被检索出来，并与原始问题一起作为上下文提供给 Phi-4（通过 Ollama 本地运行）。
System B — Graph Only
系统 B — 仅图谱检索
Where System A treats documents as collections of passages, this approach treats them as collections of facts. Rather than retrieving text, the system retrieves structured knowledge — concepts and the relationships between them — stored in a graph database. I built the graph by extracting entity-relationship triples from the same source documents.
如果说系统 A 将文档视为段落的集合，那么这种方法则将它们视为事实的集合。系统检索的不是文本，而是存储在图数据库中的结构化知识——即概念及其之间的关系。我通过从相同的源文档中提取实体-关系三元组来构建该图谱。