|
AI’s Next Bottleneck Is Deployment. (Sponsored)
 |
Turning new models into systems that work inside real customer operations is still hard.
That gap is creating demand for engineers who can move between code, customer context, and production outcomes. Enter: the forward deployed engineer.
The free State of FDE Jobs 2026 report maps the emerging labor market around this work.
Imagine an AI-based retrieval system pointed at five years of your team’s engineering documents, including design docs, incident postmortems, and architecture decision records. Someone asks which service owns the payments retry logic, and a pretty accurate and well-cited answer is provided by the system. However, when someone asks which failure causes recur most often across all the postmortems, the quality of the answer goes down.
Depending on the setup, the response might list a handful of incidents that happen to use the word recurring, but we don’t get any idea of the underlying pattern from the answer. In other words, the reason for asking the question is not fulfilled.
Both questions can look similar from the outside. Architecturally, however, they are opposites:
The first has an answer that can be found in a specific document, which is precisely what similarity search was built for.
The second has an answer that shows up only after the entire collection has been surveyed and understood. This requires a completely different retrieval mechanism.
GraphRAG was designed to handle the second kind of questions, and we are going to learn more about it in this article. Here’s what we will cover:
How standard RAG retrieval works, and where it reaches its limit
Knowledge graphs, and how one gets built from ordinary documents
The GraphRAG indexing pipeline
Community detection and hierarchical summaries
Local search and global search
Cost, latency, and maintenance tradeoffs
When standard RAG remains the better option
Agentic RAG
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.
 |
Retrieval Basics
Standard RAG (Retrieval Augmented Generation) depends on a compact pipeline.
We take a collection of documents, slice each one into chunks of a few hundred to a few thousand tokens, and pass every chunk through an embedding model. The embedding model returns a vector, which is basically a long list of numbers standing in for the meaning of that text. Chunks with related meanings produce vectors that sit close together in the same numeric space. All of those vectors go into a vector index.
At query time, the same treatment applies to the question. The question also becomes a vector, the index returns the handful of chunk vectors closest to it, and the original text of those chunks gets placed into the prompt alongside the question. The language model then generates an answer from the supplied text.
 |
The whole design rests on one simple assumption, which is that text answering a question would resemble that question. For a large share of queries, this assumption holds up well. For example, a question like “Which service owns the payments retry logic” contains the same vocabulary as the architecture decision record where that ownership was recorded. The vectors land near each other, retrieval returns the right document, and the citation points somewhere a reader can actually verify.
Similarity Limits
This assumption about questions being similar to the answers holds for a specific class of questions, but it is by no means a universal thing.
Microsoft’s GraphRAG documentation distinguishes local queries from global queries. A local query has an answer that resembles the query and lives inside a small number of text regions, which covers most who, what, when, and where questions. A global query requires reasoning across large portions of a dataset, or across all of it.
Our two example questions from earlier land on opposite sides of that line. The question “Which service owns the retry logic” is local. However, the question “Which failure causes recur most often across all postmortems” is global.