Chunking strategy, hybrid search, reranking budget — retrieval design is a stack of small decisions that everyone argues about and almost nobody measures. This playbook makes each decision at the architecture level, then commits every one of them to a retrieval eval before it enters the design.
Tier: Core · Hands-on time: ~35 minutes
Phase-1 posting research names chunking, embeddings, hybrid search, and reranking explicitly as required skills, and this track owns the decision layer: which strategy fits which document types, what latency and cost a reranker is allowed to spend, and how the choice gets validated. The validation rule is absolute — retrieval decisions are settled by Ragas-class metrics on a golden set, not by whoever describes their intuition most confidently, and not by an AI's plausible-sounding defaults.
Classify your corpus before choosing anything. Chunking that works for FAQs mangles contracts. List document types, their structure (headings, tables, clause numbering), and typical question shapes against each. This classification, not a global default, drives the chunking decision.
Enumerate candidate strategies per document type with AI — as hypotheses. Anita's orbit-docs-copilot corpus mixes API references, how-to guides, and release notes:
For a docs copilot over three content types — API reference pages
(structured, table-heavy), how-to guides (narrative with code blocks),
and release notes (short, dated) — propose 2 chunking strategies per
type with the failure mode each strategy risks for that type.
Then propose: (a) when hybrid (BM25 + vector) search would beat pure
vector for this corpus, with the query shapes that decide it, and
(b) what evidence would justify adding a reranker, given each reranked
query adds latency and per-query cost.
Frame everything as testable hypotheses with the metric that would
confirm or kill each one. Do not present any option as best practice.
Design the experiment matrix, small. Two chunking variants, pure-vector vs hybrid, reranker on/off — run only the cells that could change the design. Each cell gets scored on the same golden set with the same metrics (context precision, context recall, faithfulness of downstream answers).
Run the evals and read them like an architect. A variant that wins recall by 2% but doubles index size and complicates ingestion may still lose. Record metric deltas next to their cost: index size, ingestion complexity, query latency, per-query spend.
Give the reranker a budget, not a blessing. If reranking wins on the golden set, write its latency and cost budget into the design explicitly — which queries get reranked (all? low-confidence only?) and what the p95 latency target absorbs.
Write the retrieval design doc from results. One section per decision: chosen strategy, the eval numbers that chose it, the losing option's best number, and the trigger that reopens the decision (new document type, golden-set drift, latency regression).
Wire the golden set into the future. The same set that made these decisions becomes the regression gate for the next change — hand the eval harness to the build team as part of the design (pb-33 designs that loop fully).
Client documents often cannot leave approved boundaries — extract the MSA constraints first. Build the experiment with a synthetic corpus that mirrors the client's document structure (same formats, section shapes, and table density, fake content) so chunking behavior transfers; generate it with schema-faithful synthetic-data prompts. Run embeddings and evals against client-approved tenancy (Bedrock or Azure OpenAI in the client's subscription) or fully locally — LlamaIndex with local embeddings via Ollama and Ragas offline keeps the whole loop on your machine. The decision doc and eval discipline are identical either way.
Every retrieval decision is a hypothesis until a golden-set eval settles it — AI enumerates the options, your Ragas run picks the winner, and the numbers go in the design doc next to their costs.