Lab 03 · Log Analysis with Trust Calibration

Feed a night's worth of Windows event log and syslog extracts to an LLM, collect its hypotheses and timeline — then verify every single claim against the raw lines. One root cause is real, three red herrings are planted, and the model will be confidently wrong at least once. Your RCA must show exactly where.

Meena is the on-call sysadmin for an internal HR application at a Hyderabad GCC. At 02:10 the app started throwing intermittent 502s; by 02:30 monitoring paged her; by 03:00 it recovered on its own. Her manager wants the root cause before the 10:00 review, and Meena wants to use an LLM to get there fast — the logs are long and the clock is short.

The trap: these logs contain one true cause and several excellent decoys, including the single most seductive wrong answer in Windows administration. An LLM pattern-matches to the decoys with total confidence. The lab's product is an RCA that cleanly separates AI hypothesis from verified finding, and documents where the model misled you — because that separation is what makes AI log analysis fast and safe.

Time & tier

~75 minutes hands-on · Core tier (maps to module M2). $0 — any free AI chat plus a text editor. All log data is inline and synthetic (server names and IPs are fictional).

Setup

Save the two extracts below as appsrv-events.txt and proxy-syslog.txt. The Windows extract is from HR-APPSRV-01 (Windows Server 2022, IIS app + SQL Express, app logs and backups on D:). The syslog is from the Linux reverse proxy in front of it.

=== HR-APPSRV-01 · System/Application event extract (UTC+05:30) ===
Aug 04 22:15:03 EventID 7036 Service Control Manager: "Windows Update" entered running state
Aug 04 23:02:11 EventID 36887 Schannel: A fatal alert was received from the remote endpoint. TLS alert code: 46
Aug 04 23:47:52 EventID 36887 Schannel: A fatal alert was received from the remote endpoint. TLS alert code: 46
Aug 05 00:31:19 EventID 4625 Security: An account failed to log on. Account: svc-hrscan  Source: 10.20.8.44  Failure reason: bad password
Aug 05 00:31:24 EventID 4625 Security: An account failed to log on. Account: svc-hrscan  Source: 10.20.8.44  Failure reason: bad password
Aug 05 00:31:29 EventID 4625 Security: An account failed to log on. Account: svc-hrscan  Source: 10.20.8.44  Failure reason: bad password
Aug 05 01:45:40 EventID 19 WindowsUpdateClient: Installation successful: KB5062832 (2026-07 Cumulative Update)
Aug 05 01:47:02 EventID 1074 User32: Restart initiated by NT AUTHORITY\SYSTEM: Operating system upgrade (planned)
Aug 05 01:52:36 EventID 6013 EventLog: system uptime is 94 seconds
Aug 05 01:53:10 EventID 7036 Service Control Manager: "World Wide Web Publishing Service" entered running state
Aug 05 01:53:12 EventID 7036 Service Control Manager: "SQL Server (SQLEXPRESS)" entered running state
Aug 05 02:00:01 EventID 129 TaskScheduler: Task "NightlyFullBackup-HRDB" launched (D:\backups\)
Aug 05 02:08:44 EventID 2013 srv: The D: disk is at or near capacity. You may need to delete some files.
Aug 05 02:09:58 EventID 17053 SQLEXPRESS: OS error 112 (There is not enough space on the disk) encountered.
Aug 05 02:10:05 EventID 3041 SQLEXPRESS: BACKUP failed to complete the command BACKUP DATABASE HRDB.
Aug 05 02:10:31 EventID 1309 ASP.NET: Event code 3005. Exception: System.IO.IOException: There is not enough space on the disk. Path: D:\applogs\hrapp\audit.log
Aug 05 02:10:33 EventID 5011 WAS: A process serving application pool 'HRAppPool' suffered a fatal communication error.
Aug 05 02:11:00 EventID 5002 WAS: Application pool 'HRAppPool' is being automatically disabled due to a series of failures.
Aug 05 02:14:22 EventID 4625 Security: An account failed to log on. Account: svc-hrscan  Source: 10.20.8.44  Failure reason: bad password
Aug 05 02:26:37 EventID 7036 Service Control Manager: "HRApp Cleanup Agent" entered running state
Aug 05 02:27:12 INFO CleanupAgent: purged 41 GB from D:\backups\ (retention 7d exceeded: retention job had been failing since Jul 29)
Aug 05 02:29:48 EventID 5211 WAS: Application pool 'HRAppPool' was re-enabled by health monitor.
Aug 05 02:30:04 EventID 1309 ASP.NET: Event code 3005 cleared. Application restarted normally.
Aug 05 03:15:07 EventID 36887 Schannel: A fatal alert was received from the remote endpoint. TLS alert code: 46

=== proxy-01 · /var/log/nginx extract (UTC+05:30) ===
Aug 05 01:47:10 proxy-01 nginx: [error] upstream 10.20.4.15:443 connect() failed (111: Connection refused) while connecting to upstream, client 10.20.9.31
Aug 05 01:47:41 proxy-01 nginx: [error] upstream 10.20.4.15:443 connect() failed (111: Connection refused), client 10.20.9.87
Aug 05 01:53:20 proxy-01 nginx: [info] upstream 10.20.4.15:443 recovered, passive health check OK
Aug 05 02:10:35 proxy-01 nginx: [error] upstream 10.20.4.15:443 timed out (110: Connection timed out) while reading response header, client 10.20.9.44
Aug 05 02:10:36 proxy-01 nginx: 10.20.9.44 - "POST /hrapp/api/leave HTTP/1.1" 502
Aug 05 02:11:02 proxy-01 nginx: [error] no live upstreams while connecting to upstream, client 10.20.9.51
Aug 05 02:11:02 proxy-01 nginx: 10.20.9.51 - "GET /hrapp/home HTTP/1.1" 502
Aug 05 02:14:48 proxy-01 nginx: 10.20.9.62 - "GET /hrapp/home HTTP/1.1" 502
Aug 05 02:22:19 proxy-01 nginx: 10.20.9.44 - "GET /hrapp/api/leave HTTP/1.1" 502
Aug 05 02:29:55 proxy-01 nginx: [info] upstream 10.20.4.15:443 recovered, passive health check OK
Aug 05 02:30:20 proxy-01 nginx: 10.20.9.44 - "POST /hrapp/api/leave HTTP/1.1" 200
Aug 05 03:40:11 proxy-01 nginx: 10.20.9.102 - "GET /hrapp/home HTTP/1.1" 200

Incident window as reported by monitoring: 502s from 02:10 to 02:30. Known context you may give the AI if asked: KB5062832 was approved for this ring last week; svc-hrscan is a scanner service account on a different host; the Schannel alerts have appeared for months.

Guided steps

Write your own 3-line hypothesis first. Before any AI contact, skim both extracts and write in rca.md under "Pre-AI hypothesis": what you think broke, when, and why. This calibrates you — you will grade yourself against the key too.
Get the AI's full analysis. Paste both extracts with: "You are assisting a root-cause analysis. Incident: HTTP 502s on hrapp 02:10–02:30. From these logs give: (a) a reconstructed timeline of significant events, (b) ranked root-cause hypotheses with confidence, (c) anomalies worth investigating, (d) the evidence line for every claim." Save the raw output as ai-analysis.md untouched.
Verify the timeline line-by-line. For each event the AI placed on its timeline, find the actual log line and check the timestamp and the wording. Mark each timeline entry [verified] or [not in logs]. Expected output: at least one entry that is subtly wrong or invented — models routinely shift timestamps or merge two events into one.
Interrogate the top hypothesis. Whatever the AI ranked #1, ask it: "What evidence contradicts this hypothesis? What would we expect to see in these logs if it were true, that we do not see?" Note whether it finds the contradiction or defends itself. There is a decoy here with beautiful surface correlation — check whether the AI fell for it, and record the exchange.
Run the discriminating test on the two leading causes. For the top two hypotheses, identify the single log line that separates them (which came first, what does the SQL error actually say, what does the recovery event coincide with). Write the elimination logic explicitly in rca.md: "If X were the cause, the 01:47 proxy errors would/would not…".
Chase the anomalies honestly. The AI will likely flag the 4625 failed logons and the Schannel alerts. For each: is it related to this incident window? Is it still worth a follow-up ticket? Classify each anomaly as related, unrelated-but-actionable, or noise, with the reasoning.
Explain the recovery. A verified RCA explains why the incident ended at 02:29, not just why it began. Find the exact mechanism in the logs — it also reveals the deeper cause underneath the immediate one.
Write the final RCA. Structure rca.md: Incident summary · Verified timeline (cited lines only) · Root cause (immediate + underlying) · What the AI got right · Where the AI misled (verbatim quote of its wrong claim + the log line that refutes it) · Follow-up actions. Expected length: one page.

Warning

"The patch did it" is the most seductive sentence in Windows RCA — a KB installed and a reboot happened hours before almost every incident ever. Correlation with an update is a hypothesis to discriminate, never a conclusion to publish.

Verify

ai-analysis.md preserved verbatim; rca.md contains your pre-AI hypothesis, unedited after the fact.
Every timeline entry in the final RCA cites an actual log line; every [not in logs] claim from the AI is listed in the misled section.
The RCA names an immediate cause and an underlying cause, and they are different things.
The 01:47 proxy errors and the 02:10 proxy errors are attributed to two different mechanisms.
Each of the three anomaly families (failed logons, Schannel, update/reboot) is classified with reasoning.
The recovery at 02:29 is explained by a specific logged event, not "it came back".

Self-grading rubric (pass = 7/10): correct immediate root cause (0–2) · underlying cause found (0–2) · decoy explicitly refuted with the discriminating line (0–2) · AI's wrong claim quoted and countered (0–2) · anomalies triaged correctly (0–1) · recovery mechanism identified (0–1).

▸Answer key — root cause, decoys, and the expected AI failureoptional deep dive — click to expand

Immediate root cause: the 02:00 NightlyFullBackup-HRDB task filled D:. Evidence chain: 02:08:44 disk-near-capacity (2013) → 02:09:58 SQL OS error 112 not enough space → 02:10:31 ASP.NET IOException writing D:\applogs\hrapp\audit.log → 02:10:33/02:11:00 app pool fatal error and auto-disable → proxy "no live upstreams" 502s from 02:10:35. The app died because it could not write its audit log, and WAS rapid-fail-protection disabled the pool.

Underlying root cause: the backup retention/cleanup job had been silently failing since Jul 29 — stated in the 02:27:12 CleanupAgent line ("retention job had been failing since Jul 29", 41 GB purged). Seven days of un-pruned full backups is why one more nightly backup tipped the disk. An RCA that stops at "disk full" misses the real fix.

Recovery mechanism: CleanupAgent freed 41 GB at 02:27, WAS health monitor re-enabled the pool at 02:29:48, proxy health check recovered 02:29:55. Not "self-healed" — a specific, logged sequence.

The prime decoy (expected AI confident-wrong): KB5062832 at 01:45 + planned reboot at 01:47. Surface correlation is perfect and models frequently rank it #1. Refutation: the 01:47–01:53 proxy connection refused errors are the reboot window itself (services down), explicitly recovered at 01:53:20; the app then served traffic cleanly for ~17 minutes; the 02:10 failure signature is disk-space IO errors, not update-related. If the patch had broken IIS, the 01:53 recovery and the space-error chain would both be inexplicable.

Secondary decoys: (1) svc-hrscan 4625 bursts — wrong password from a scanner host; three at 00:31, one at 02:14; unrelated to the outage window's mechanism but unrelated-but-actionable (stale credential on 10.20.8.44 deserves a ticket; also a lockout risk). An AI calling this a "brute-force attack during the incident" is over-reading four events. (2) Schannel 36887 alerts — months-old ambient noise, present before and after; classify noise (with an optional hygiene follow-up). (3) SQL BACKUP failed (3041) — a symptom of the full disk, not a cause; an RCA blaming "database backup failure" has the arrow backwards.

Common invented-timeline errors to catch: merging the 01:47 and 02:10 proxy errors into one continuous outage; claiming the app pool crashed "due to the reboot"; asserting monitoring paged at a time that appears in no log.

Scoring: disk-full chain cited = 2 · retention-failure underlying cause = 2 · KB decoy refuted with the 01:53 recovery + 17-minute clean window = 2 · AI wrong claim quoted verbatim with counter-line = 2 · 4625/Schannel/3041 correctly triaged = 1 · recovery sequence = 1.

Key Takeaway:

LLMs turn hours of log reading into minutes of hypothesis generation — and every hypothesis is guilty until a raw log line proves it. The RCA that separates "the AI suggested" from "the logs show" is faster than working alone and safer than trusting alone.

© Anvi Innovate · BXUP — Be Exponentially Upskilled · BXUP is an independent learning platform, not affiliated with or endorsed by AWS, Microsoft, Google, NVIDIA, IBM, Salesforce, Databricks, Oracle or Anthropic. All trademarks belong to their respective owners. · bxup.in