Log Forensics: Distill Thousands of Log Lines into a Timeline

Tier: Core · Workflow area: Debugging

Long-context models are genuinely good at exactly this — extraction and correlation over bulk semi-structured text — if you treat them as an extraction engine with a schema, not an oracle you ask "what went wrong?". The output is a verified incident timeline plus an anomaly set that turns "it happens sometimes" into "it happens when X".

When to use

• •
  Intermittent failures ("about twice a day, no pattern we can see") with logs but no reliable repro.
• •
  Post-incident reconstruction across multiple services' logs.
• •
  Triaging a noisy error budget: which of these 400 distinct warnings actually precede failures?

Skip it when a log search tool answers the question in one query — grep/Loki/CloudWatch Insights first, LLM when the question needs correlation across lines, not lookup.

Prerequisites

• •
  Log extracts around several failures (±5 minutes each side), plus one comparable healthy window as a control.
• •
  Whatever structure exists: request IDs, correlation IDs, worker/pod names. If you have none, this playbook still works but delivers less — and adding correlation IDs becomes finding #1.
• •
  A long-context model (current frontier chat models handle tens of thousands of lines) — or the chunked variant in Tool variants.

The workflow

1. 1.

   Cut disciplined slices, not the whole firehose. Pull each failure window with context, pre-filtered mechanically:

   Copy

   # 3 failure windows + 1 healthy control, all services interleaved by ts
   grep -h "" api.log worker.log payments.log \
     | awk '$1 >= "12:41:00" && $1 <= "12:52:00"' | sort > window1.log
   

   Mechanical reduction (time windows, dropping DEBUG spam categories you already trust) is cheap and lossless-enough; semantic reduction is the model's job.

2. 2.

   Extract with a schema, per window. The prompt shape that keeps it honest:

   Copy

   These are interleaved logs (api, worker, payments) for a window ending in
   a failed checkout at 12:51:47 (request id req-8834). Build:
   1. TIMELINE: every event relevant to req-8834 and to shared resources it
      touched (DB pool, redis, payment gateway), as
      `ts | service | event | verbatim log line`. Quote lines EXACTLY.
   2. ANOMALIES: anything unusual vs normal operation in this window -
      repeated retries, latency jumps, restarts, out-of-order events,
      gaps > 5s in a normally chatty sequence. Same quoting rule.
   3. Do not conclude a root cause yet.
   Unquoted claims will be discarded.
   

3. 3.

   Spot-check the quotes. Grep three or four quoted lines against the raw file. Fabricated or "cleaned-up" quotes at this stage poison everything downstream; if quotes don't check out verbatim, tighten the instruction and rerun.

4. 4.

   Correlate across windows — failure vs control. Now the real question:

   Copy

   Here are timelines/anomaly sets for 3 failure windows and 1 healthy
   control (attached). What is present in all failure windows and absent
   from the control? List candidate discriminators ranked by consistency,
   each with its supporting quotes from each window. Explicitly list
   anomalies that appear in the control too - those are background noise.
   

   A real answer looks like: "in all 3 failures, worker logs redis reconnecting 2-9s before the checkout starts; control window has no reconnects."

5. 5.

   Convert the discriminator into a check, then a repro. Verify at scale with a mechanical query (Loki/CloudWatch/grep over weeks of history): does redis reconnecting within 10s actually precede failures at a rate far above baseline? If yes, you have the trigger condition — hand it to pb-11 to build the deterministic repro (kill the redis connection mid-checkout in a test) and fix from there.

6. 6.

   File the byproducts. Log-forensics runs always surface secondary findings: missing correlation IDs, log-level abuse, a service logging secrets. Ticket them — next incident's forensics gets cheaper.

Tool variants

• Long-context chat model (frontier tier): paste or attach windows directly; simplest path for up to ~50k lines total.
• Chunked pipeline for bigger volumes or smaller models: run step 2 per window (or per 5k-line chunk) separately, then feed only the compact timelines/anomaly sets into step 4 — the correlation step needs the digests, not the raw bulk.
• lnav / log tool + assistant: use lnav's filtering and SQL to build the slices interactively, assistant for the extraction prompts; strongest combo when you don't yet know which window matters.

Verification

• •
  Every timeline entry spot-checked back to a verbatim raw line (minimum 3-4 per window).
• •
  The discriminator validated mechanically over a longer history with real numbers ("precedes 11/12 failures, base rate 0.4% of healthy minutes").
• •
  The control-window comparison exists — anomaly lists without a control are astrology; everything looks anomalous in a window you're staring at.
• •
  Trigger condition reproduced deterministically (or explicitly recorded as not-yet-reproducible with the discriminator as monitoring alert instead).

Pitfalls

• Asking "what caused this?" on the first pass. The model will oblige with a fluent narrative assembled from vibes. Extraction first, correlation second, causation only after mechanical validation.
• No healthy control. The single most common methodological hole — retries and warnings that also occur constantly in healthy traffic get promoted to villains.
• Silent truncation. Very large pastes can be clipped; ask "quote the last log line you received" and check it is your last line.
• Trusting summarized quotes. Models normalize whitespace, fix typos, merge lines. Verbatim-or-discard is the rule that keeps the timeline admissible.

Restricted-environment variant

Logs are the most regulated artifact here: they routinely contain personal data (DPDP-relevant in India), tokens, and internal topology. Options in order: run forensics inside the client's tenancy (Bedrock/Azure OpenAI-backed assistant reading logs that never leave their cloud); sanitize before pasting — a sed pass mapping user IDs/emails/hosts to stable placeholders (stable mapping matters: user_A must stay user_A across lines or correlation dies); or the chunked local-model pipeline (Ollama) — extraction-with-schema is well within local-model range, and the mechanical validation step carries the rigor regardless.

Success criteria

• •
  An intermittent bug acquired a named, validated trigger condition — "when redis reconnects mid-checkout" — where before there was "sometimes".
• •
  Timeline and anomaly set produced in under an hour for a volume you could not have read in a day.
• •
  The discriminator's hit rate was measured over full history, not asserted from three windows.
• •
  At least one logging-hygiene improvement ticketed as a byproduct.