Lab 03 · ADR Factory with a Red-Team Pass

Anita's team at Orbit made five real architecture decisions in a 41-minute call about ingestion eventing. As usual, none were written down. Six months from now someone will ask "why SQS and not Kafka?" and the answer will be a shrug. The ADR factory pattern — AI drafts, human decides, adversary attacks — makes capturing decisions cheaper than losing them.

Time & tier

~75 minutes hands-on · Core tier (maps to module M3). Cost: ₹0 — design-artifact lab; any free-tier chat assistant or a local Ollama model.

Setup

Create orbit-adrs/ with the transcript below as transcript.md, plus an empty adr/ directory. Use MADR-style structure for every ADR: Title · Status · Context · Decision · Options considered · Consequences · Revisit when.

[Orbit #arch-sync — ingestion eventing — 26 Jun, 41 min, edited]

Anita: Recap: orbit-docs-copilot ingestion is S3-upload driven. The embed
  worker fires straight off the S3 notification. Two incidents last month:
  a 4,000-file bulk upload throttled us, and we dropped events during a
  worker deploy. I want a buffer between upload and embed.
Karthik (dev lead): Kafka. We'll need streaming eventually — click
  analytics, audit feeds. MSK gives us replay and multiple consumers.
Meera (SRE): Who operates it? We are four people. MSK is still brokers,
  partitions, rebalancing pain. What does ingestion actually need?
Anita: Durability, retry, backpressure. One consumer today. Fan-out is
  speculative — nothing on the roadmap needs a second consumer this year.
Karthik: EventBridge then? Native S3 integration, no brokers at all.
Meera: EventBridge is a router. We want a work queue — per-message retry
  and a dead-letter path. That's SQS.
Anita: Agreed. SQS as the ingestion buffer. Kafka becomes a re-decision
  if a second consumer or a replay requirement actually lands — I'll
  write the exit trigger down so this doesn't harden into dogma.
Karthik: FIFO or standard? A tenant's docs should embed in order.
Meera: Should they? Chunks are independent rows. The only real hazard is
  two revisions of the same doc embedding out of order.
Anita: Handle that at write time — the worker compares revision ids and
  refuses to overwrite newer with older. Idempotent, revision-aware
  consumer means standard queue is fine and we keep the throughput.
Meera: Poison messages. A corrupt PDF crashed the worker twice in May
  and it just retried forever.
Anita: Dead-letter queue. Five receives, then DLQ. Alarm on DLQ depth
  greater than zero; weekly redrive review. Meera owns the alarm.
Karthik: What's the message body? Today the worker parses the raw S3
  event shape. Sales wants a SharePoint connector next year — that
  coupling leaks everywhere.
Anita: Wrap it. Versioned envelope: schema_version, tenant_id, doc_id,
  revision, source, object_uri. Worker validates the envelope and sends
  unknown major versions to the DLQ. No S3-specific fields in the body.
Meera: One more — the upload API waits for embedding to finish before
  returning 200. That's why bulk uploads time out.
Anita: That dies today. Upload returns 202 once the queue write commits.
  Embedding is async; the doc registry tracks status.
Karthik: Sales will ask why an upload "finished" but search misses it.
Anita: Status field — uploaded, embedding, searchable — surfaced in the
  UI. Okay: I owe the ARB five ADRs from this call. Drafts by Friday.

Guided steps

Step 1 — Extract the decision list (10 min)

Read the transcript yourself and list every decision actually taken (not merely discussed) as one line each: decision, who pushed back, what alternative lost. Then ask your assistant to do the same extraction, and diff the two lists. There are five decisions; keep only ones the transcript supports — AI extraction routinely promotes discussion points ("Kafka someday") into decisions.

✅ Checkpoint: a reconciled list of exactly 5 decisions, each traceable to transcript lines.

Step 2 — Draft five ADRs (20 min)

For each decision, prompt the assistant with the transcript plus the MADR skeleton and this instruction: "Draft the ADR. Context and options must come only from the transcript. Where the transcript gives a rationale, quote it. Leave Status as 'proposed'. Do not invent benchmarks, limits, or prices." Save as adr/0001-…md through adr/0005-…md. Read each draft against the transcript before moving on — the classic failure is the AI backfilling plausible-but-unsaid rationale, which turns your decision record into fiction.

✅ Checkpoint: five ADRs on disk, every Context/Options claim traceable to the transcript, all statuses "proposed".

Step 3 — Pick the highest-stakes ADR and red-team it (20 min)

The backbone choice (SQS over Kafka/EventBridge) carries the most irreversibility and the most future traffic. Attack it with a fresh conversation — do not reuse the drafting chat, which is anchored to agree with itself:

You are a hostile principal engineer reviewing this ADR before the ARB.
Attack it: What breaks at 100x document volume? What does the "second
consumer" future actually cost under this choice vs the losing options?
Which stated rationale is weakest? What failure mode is unhandled? What
would make this decision look foolish in 18 months? Be specific; no
praise; do not rewrite the ADR — produce numbered objections only.

Triage the objections into three buckets in redteam-0001.md: accept (changes the ADR), acknowledge (goes into Consequences as a known risk), reject (wrong or out of scope — say why). Verify any objection that leans on a service capability or quota claim against vendor docs before accepting it.

✅ Checkpoint: redteam-0001.md with ≥6 objections triaged into accept/acknowledge/reject, each with a one-line reason.

Step 4 — Revise the attacked ADR (15 min)

Fold accepted objections into the decision text, acknowledged ones into Consequences, and sharpen "Revisit when" into concrete exit triggers (e.g., a second consumer with a committed delivery date, a replay requirement, sustained throughput beyond the queue's comfort zone — expressed as observable conditions, not vibes). Keep the original draft as adr/0001-…-v1.md so the before/after diff is visible. Set the revised ADR's status to "accepted".

✅ Checkpoint: revised ADR 0001 plus preserved v1; diff shows substantive changes driven by the red team.

Step 5 — Consistency pass across the set (10 min)

Ask the assistant to cross-check all five ADRs for contradictions (does the envelope ADR agree with the DLQ ADR on how invalid messages die? does the async-ingestion ADR reference the status field consistently?) and fix what it finds — after confirming each reported contradiction is real.

✅ Checkpoint: a noted-and-fixed (or explicitly none-found) contradiction list appended to redteam-0001.md.

Verification

• Decision coveragefive ADRs matching the five decisions the transcript actually contains (compare with the answer key after finishing).
• Traceabilityspot-check three Context claims; each maps to transcript lines, none invented.
• Red-team depth≥6 specific objections, at least one accepted into the ADR and at least one rejected with a documented reason.
• Before/afterv1 and revised ADR 0001 differ materially: exit triggers are observable conditions.

Completion artifact

The orbit-adrs/ folder: 5 ADRs, the preserved v1, redteam-0001.md. Self-grading rubric (pass = 7/9):

• Extraction accuracy (0–2)five correct decisions, no promoted discussion points = 2.
• ADR fidelity (0–3)no invented rationale across the set = 3; one instance = 1.
• Red-team quality (0–2)objections are specific and triaged with reasons = 2.
• Revision quality (0–2)accepted objections visibly changed the decision text and exit triggers = 2.

Client-safe variant

Real design calls are full of client names and commercial context. Under a restrictive MSA: run this lab's synthetic transcript as training, then for real work either use the client-approved tenancy, or pre-redact transcripts (names → roles, systems → generic labels) before prompting, or run extraction and drafting on a local model via Ollama. The ADRs themselves are usually client deliverables — the raw transcript and chat history are what must not leak.

Stuck?

• •
  The AI finds seven or eight "decisions": require a transcript quote where someone commits ("Agreed", "that dies today") for each; discussion without commitment is context, not a decision.
• •
  Red-team output is generic ("consider scalability"): demand numbered objections tied to specific ADR sentences, and give it the losing options explicitly.
• •
  Red team asserts a quota or capability you can't confirm: park it as "acknowledge — verify against vendor docs" rather than accepting it into the decision.
• •
  ADRs keep drifting in format: paste ADR 0001 as the style exemplar into every subsequent drafting prompt.

Answer key (don't read before attempting)

The five decisions: (1) SQS as the ingestion buffer — over Kafka/MSK (ops burden, speculative fan-out) and EventBridge (router, not a work queue); exit triggers: real second consumer or replay requirement. (2) Standard queue + idempotent, revision-aware consumer — over FIFO; ordering hazard handled at write time via revision comparison. (3) DLQ after five receives, alarm on depth > 0, weekly redrive review, Meera owns the alarm. (4) Versioned event envelope (schema_version, tenant_id, doc_id, revision, source, object_uri); unknown major versions to DLQ; no source-specific fields. (5) Asynchronous ingestion — upload returns 202 on queue-write commit; doc-registry status field (uploaded → embedding → searchable) surfaced in the UI.