Summarize a Long Document and Verify the Summary

Tier: Foundation · Hands-on time: ~25 minutes

This playbook produces a summary you can defend, using the claim-audit method: every claim in the summary is traced back to the source before you ship it.

When to use

• •
  A 10+ page document (spec, contract, RFC, audit report, research paper) that others expect you to have read.
• •
  You need to brief someone else and will be held accountable for accuracy.
• •
  You are triaging many documents and deciding which deserve a full read.

Skip AI entirely for documents under two pages — reading them yourself is faster than verifying a summary.

Prerequisites

• •
  Access to any current general-purpose AI assistant with a context window large enough for your document (most current models handle 100+ pages of plain text).
• •
  The document as text, Markdown, or PDF. If it is a scanned image PDF, convert or OCR it first.
• •
  5 minutes to skim the source yourself first — you cannot audit claims about a document you have never opened.

The workflow

1. 1.

   Skim the source first (2–3 minutes). Read the title, headings, first and last sections. Note two or three facts you now know are in the document. These become your canary checks later.

2. 2.

   Request a structured summary with rules. Paste or attach the document with this prompt:

   Copy

   Summarize the attached document for a technical audience.
   
   Rules:
   - Structure: (1) one-sentence purpose, (2) 5-8 key points, (3) decisions or
     obligations stated in the document, (4) numbers, dates, and named parties,
     (5) what the document explicitly does NOT cover.
   - Use only information present in the document. Do not add background
     knowledge, do not infer intent.
   - After each key point, add the section or page it came from in brackets.
   - If something is ambiguous in the source, say "ambiguous in source"
     instead of resolving it yourself.
   

3. 3.

   Run the claim audit. In the same conversation, send:

   Copy

   Now list every factual claim in your summary as a numbered list. For each
   claim, quote the exact sentence(s) from the document that support it,
   verbatim. If you cannot find a supporting quote, write "NO DIRECT SUPPORT"
   for that claim.
   

4. 4.

   Spot-check the quotes. Use Ctrl+F / Cmd+F in the original document to verify at least three quoted sentences exist verbatim, including your canary facts from step 1. Any claim marked NO DIRECT SUPPORT gets deleted from the summary or verified by you manually.

5. 5.

   Fix and finalize. Ask the model to regenerate the summary excluding unsupported claims, then paste the result into your own words where tone matters. You own the final text, not the model.

Tool variants

• Chat assistant with file upload (any major vendor): attach the PDF directly; the structured prompt works unchanged. Best for one-off documents.
• AI-enabled editor or IDE assistant: open the document in your editor and run the same prompts against the open file — useful when the document is Markdown or code-adjacent (RFCs, design docs) already in a repo.
• Local model (via Ollama or similar): works for confidential documents; smaller local models need the document split into sections of a few thousand words each, summarized per section, then merged. See the restricted variant below.

Verification

• •
  At least 3 quoted support sentences found verbatim in the source via text search.
• •
  Your 2–3 canary facts from the initial skim appear correctly in the summary.
• •
  Every number, date, and named party in the summary matches the source exactly — check each one; these are the most common hallucination sites.
• •
  The "does NOT cover" section is accurate — a summary that overclaims scope is worse than a shorter one.

Pitfalls

• Auditing in a fresh chat. Run the claim audit in the same conversation with the document still attached; otherwise the model audits from memory and invents quotes.
• Trusting paraphrased "quotes." If Ctrl+F does not find the sentence, it is not a quote. Near-matches count as failures.
• Very long documents silently truncated. If the model's summary ignores the last third of the document, it may not have received it all — ask "What is the final section of the document about?" and check.
• Letting the model resolve ambiguity. A confident-sounding interpretation of a vague contract clause is a legal risk, not a summary.

Restricted-environment variant

For client documents or anything under NDA:

• •
  Strip client names, project codenames, financial figures, and personal data before pasting into any hosted tool — replace with placeholders (CLIENT_A, AMOUNT_1) and keep a local mapping file.
• •
  Better: run the whole workflow on a local model with Ollama (see pb-07). An 8B-class instruct model handles section-by-section summarization well; audit claims per section.
• •
  Never upload signed contracts or personal data to a consumer AI tool account; use only your organization's approved enterprise tenant if one exists.

Success criteria

• •
  You produced a summary in under 30 minutes for a document that would take 2+ hours to read fully.
• •
  Every claim in the shipped summary is either quote-backed or verified by you personally.
• •
  A colleague who read the full document agrees the summary contains no false or invented statements.
• •
  You can state, from the summary alone, what the document does not cover.