GraphRAG & Agentic Retrieval — When Relationships Matter

Tier: Advanced · Hands-on time: ~35 minutes

GraphRAG has hyperscaler-canon status: Google's Architecture Center publishes a dedicated GraphRAG reference architecture alongside its managed and custom RAG patterns. That legitimacy is precisely why this playbook exists — the pattern is real, the demos are seductive, and the cost and complexity are routinely understated. The architect's job is to weigh vector-plus-graph retrieval and multi-hop agentic retrieval against the questions users actually ask, not against the questions the demo asks.

When to use

• The corpus is relationship-shapedorg structures, bills of materials, regulatory cross-references, contract clause hierarchies — and users ask questions that traverse those relationships.
• Plain RAG evals show a specific failure class: multi-hop questions ("what does policy A imply for supplier B's subcontractors?") retrieve fragments but miss the connecting structure.
• An agentic-retrieval proposal is on the table and needs a sober cost/latency/complexity assessment before it enters the design.
• Skip AI when your eval failures are single-hop retrieval quality — fix chunking and reranking first (pb-30); graph machinery will not rescue bad basic retrieval.

Prerequisites

• Eval evidence from your existing RAG showing which question classes fail — this playbook starts from measured failure, not enthusiasm.
• A sample of the relationship data (org chart export, BOM, clause map) and a sense of its churn rate.
• A chat assistant for the decision scaffold; Neo4j or an equivalent graph store only if you proceed to a spike.
• Google's GraphRAG reference architecture open as the canon comparison point.

The workflow

1. 1.

   Classify the failing questions. Sort real failed queries into single-hop (retrieval quality problem), multi-hop-over-text (may yield to agentic retrieval), and structural (requires explicit relationships — graph territory). Only the last two classes justify anything in this playbook.

2. 2.

   Make the AI argue both sides with the real question set. Vikram is evaluating GraphRAG for a BFSI regulations copilot where circulars reference and supersede each other:

   Copy

   Here are 12 real questions our regulations copilot failed (attached),
   plus a sample of the citation graph between circulars.
   
   For each question, classify: would it be answered by (a) better chunking/
   reranking, (b) agentic multi-hop retrieval over the existing vector
   index, or (c) an explicit knowledge graph traversal? Justify per question.
   
   Then give me the honest cost side for (b) and (c): graph construction
   and maintenance effort as the corpus churns weekly, added query latency
   for multi-hop agent loops, new failure modes (stale edges, extraction
   errors propagating into answers), and the operational skills required.
   
   Do not advocate. If plain RAG improvements would clear most of the list,
   say so.
   

3. 3.

   Cost the graph's lifecycle, not its demo. Entity and relationship extraction (usually LLM-driven — with its own error rate and token bill), graph refresh as sources churn, and the ongoing question of who owns extraction quality. A graph built once and never maintained answers last quarter's questions.

4. 4.

   Cost agentic retrieval per query. Multi-hop retrieval means multiple LLM calls and retrieval rounds per question: model the added latency (p95, not average) and per-query token spend against your budget from the capacity and cost work. Decide which query classes are allowed to pay it — usually a routed subset, not all traffic.

5. 5.

   Choose the smallest architecture that clears the failing classes. The honest ladder: improve base retrieval → add agentic multi-hop for the routed subset → add a graph only for the question classes that provably need explicit structure. Hybrid (vector for content, graph for relationships, results merged) is the common landing point in the reference literature — but only when class-(c) questions exist in volume.

6. 6.

   Spike before committing. Twenty class-(c) questions, a hand-built subgraph of the relevant entities, side-by-side answers versus plain RAG. If the graph path doesn't visibly win on questions users actually ask, the decision note says "revisit when question mix changes" — that is a successful outcome.

7. 7.

   Write the decision note and sketch. Question-class analysis, the spike numbers, chosen architecture with the merge point drawn, lifecycle ownership for extraction, and the revisit trigger. This artifact is what the design review sees.

Verify before you trust it

• The failing-question classification is checked by a human. Read each AI classification against the actual retrieval traces; the assistant will confidently misfile "bad chunking" as "needs a graph."
• Spike comparisons are scored on the same rubric. Same questions, same judge criteria for both paths — a graph demo evaluated on graph-friendly questions proves nothing.
• Latency numbers come from the spike, not the pitch. Measure multi-hop p95 on your stack; agent-loop latency claims in vendor and framework material are best-case.
• Extraction quality is sampled. Hand-check 30 extracted entities and edges against source documents; extraction error rate is the ceiling on answer trustworthiness and belongs in the decision note.
• Reference-architecture claims verified at the source. If the design cites the GraphRAG reference pattern, open the current Google Architecture Center page and confirm the pattern's stated scope matches your use — canon changes, and summaries drift.

Restricted-environment variant

Relationship data is often the most sensitive data in the building — org charts, supplier networks, contract hierarchies. Check the MSA before any of it reaches an external assistant; run the classification exercise on an anonymized or synthetic graph with the same shape (same entity types and edge semantics, fake names). Spikes run locally: Neo4j Community plus a local model via Ollama for extraction keeps the whole loop inside the boundary, and client-tenancy endpoints (Bedrock or Azure OpenAI in the client subscription) cover the middle ground where approved.