Stand Up a Project Knowledge Base with Citations and Refusals

Workflow area: Elicitation & Discovery · Tier: Core · Hands-on time: ~35 minutes

"Why did we descope the partner portal?" is a question that costs a new analyst three days and four awkward Slack messages — or thirty seconds against a knowledge base that answers with the decision-log entry, its date, and who made the call. The technology is now commodity: Confluence with Rovo, Notion AI Q&A, a Claude Project, or an LLM-plus-vector-store if you want to build. The craft is not the tool; it is two disciplines the defaults do not enforce. First, curation — a knowledge base seeded with stale drafts answers confidently from superseded decisions. Second, refusal — the system must say "not in the corpus" rather than improvise, because a plausible invented answer about a past decision is worse than no answer at all.

When to use — and when not to

• Use it on programs longer than three months with staff churnthe payback is every onboarding, every "didn't we already decide this?", every audit question about rationale.
• Use it when the same questions recurif you have answered "what did we agree about SSO?" three times, the fourth answer should come from the corpus with a citation.
• Use it before a knowledge handovera departing analyst's real legacy is a curated corpus that answers questions after they have left.
• Skip it for a corpus you have not curatedgrounded Q&A over a dump of drafts, duplicates and dead versions is a machine for laundering stale decisions into current-sounding answers.
• Skip it as the system of recordthe knowledge base answers questions about decisions; the decision log remains the authority. If they disagree, the log wins and the corpus needs recuration.
• Skip open-web tools for confidential programsthe corpus is the program's memory, including its politics and its pricing; it lives only in approved, access-controlled tooling.

The recipe

1. 1.

   Curate the corpus like it matters — because it is the product. Collect final (not draft) BRDs, the decision log, meeting notes, key emails promoted to notes, and the glossary. For each item record status (current/superseded) and date, and exclude superseded versions entirely. A 40-document curated corpus beats a 400-document dump every time.

2. 2.

   Load it into your grounded tool and set the behavior rules. In a Claude Project, Notion AI Q&A scope, or Confluence space with Rovo, add the documents, then pin behavior instructions:

   Copy

   You answer questions about this project using ONLY the documents
   in this knowledge base.
   
   Rules:
   1. Every answer must cite its source: document name, section or
      date, and quote the key sentence verbatim.
   2. If the corpus does not contain the answer, say exactly:
      "Not found in the project corpus" and suggest who or which
      artifact might know. Do not answer from general knowledge.
   3. If documents disagree, present both with citations and dates —
      do not pick a winner unless one explicitly supersedes the other.
   4. If a document is marked SUPERSEDED, never answer from it.
   5. For decision questions ("why did we..."), always include: the
      decision, the date, the decider, and alternatives considered
      if recorded.
   

3. 3.

   Red-team it before anyone else touches it. Ask ten questions you know the answers to, three questions the corpus cannot answer, and one where documents conflict. Good behavior looks like this:

   Copy

   Q: Why was the partner portal descoped?
   A: Decision D-014 (Decision Log, 12 Mar): "Partner portal moved
   to Phase 2 due to SSO dependency on the identity re-platform."
   Decided by S. Rao (sponsor). Alternative considered: interim
   manual onboarding (rejected — support cost).
   
   Q: What is the agreed SLA for vendor invoice approval?
   A: Not found in the project corpus. The Finance SOP referenced
   in BRD §4.2 may cover this — it is not loaded here.
   

   If it answers the unanswerable three from general knowledge, tighten the instructions and retest — refusal behavior is the acceptance criterion, not a nice-to-have.

4. 4.

   Wire in maintenance or watch it rot. Add new decision-log entries and meeting notes on a weekly cadence (a recurring 15-minute slot works), and mark superseded documents the day they are replaced. Name a corpus owner. An unmaintained knowledge base is worse than none, because people trust it while it drifts.

5. 5.

   Launch it with rules of use. Tell the team what it is for (fast cited answers, onboarding), what it is not (the system of record), and the one behavior that keeps it honest: any answer that matters gets its citation clicked before it is acted on. Track the questions that return "not found" — that list is your documentation backlog.

Make it yours

• Confluence + Rovo when the program lives in Atlassian: pages and decisions are already there; your work is curation, labels for superseded content, and the pinned behavior rules.
• Notion AI Q&A for Notion-native teams: scope Q&A to the project workspace and keep an explicit "Current vs Archive" page split so grounding cannot reach dead content.
• Claude Project for the analyst-owned corpus: fastest to stand up; instructions carry the refusal rules, and the project chat becomes a log of what people actually ask.
• LLM API + vector store when you need scale or access control: a build gives you chunk-level citations, per-user permissions, and eval-able refusal behavior — treat pinned-rule compliance as a test suite, not a hope.

Quality check

• Citations are real and load-bearingclicking a sampled citation lands on a sentence that actually supports the answer, not just a related document.
• Refusals fire on thin evidencethe three unanswerable red-team questions returned "not found," not improvisation; retest after every major corpus change.
• Superseded content cannot leaka question whose answer changed over time returns the current position, with the old one only as labeled history.
• Decision answers carry decider and date"why" questions return accountable answers, because rationale without an owner is folklore.
• The corpus has an owner and a cadencemaintenance is a named person's recurring job, and the last-updated date is visible to users.
• Access matches sensitivitythe knowledge base inherits the program's confidentiality; nobody can query documents they could not open directly.