Lab 01 · Alert-Triage Copilot With a Verification Gate

The skill this lab trains is not "summarize alerts faster." It is refusing to let an AI verdict leave your hands until each sentence is traceable to a field in the data. You will deliberately let the copilot reach a confident, wrong conclusion, catch it with the gate, and record the override in a false-conclusion catch log. That log — not the summaries — is the artifact your future self and your auditors will trust.

Time & tier

~90 minutes hands-on · Foundation tier (maps to module M1 · AI in the SOC — Triage & Investigation with Trust Calibration).

Setup

Everything runs offline on your laptop, $0, no cloud SIEM and no third-party systems. You will load a provided alert export into DuckDB (or plain grep/jq), and drive an LLM either through the local dry-run stub (no key needed) or a client-approved endpoint. Install: Python ≥ 3.10, pip install duckdb pydantic, and — only if you have a key — pip install anthropic. Vendor-neutral: the same pattern works with Security Copilot, Google SecOps + Gemini, or a local model via Ollama; we build on plain files so nobody is gated by a paid SIEM.

Create a working directory triage-lab/ and save this canned export as alerts.json. It is a five-alert slice of a single night on one host, exactly the kind of export you would pull from Sentinel, Splunk, or Elastic. One alert is deliberately under-specified so the copilot will be tempted to over-claim.

[
  {"alert_id":"A-1001","ts":"2026-07-20T02:14:07Z","rule":"Impossible travel","severity":"high","user":"j.rivera","src_ip":"203.0.113.44","host":"HR-LAP-12","entity_country":"BR","raw":"sign-in from BR 900ms after prior IN sign-in"},
  {"alert_id":"A-1002","ts":"2026-07-20T02:15:31Z","rule":"Multiple failed logons","severity":"medium","user":"j.rivera","src_ip":"203.0.113.44","host":"HR-LAP-12","entity_country":"BR","raw":"12 failed then 1 success"},
  {"alert_id":"A-1003","ts":"2026-07-20T02:22:19Z","rule":"Suspicious PowerShell","severity":"high","user":"j.rivera","src_ip":"10.4.4.12","host":"HR-LAP-12","entity_country":"IN","raw":"EncodedCommand + DownloadString observed"},
  {"alert_id":"A-1004","ts":"2026-07-20T03:02:00Z","rule":"EDR: LSASS handle access","severity":"critical","user":"SYSTEM","src_ip":"10.4.4.12","host":"HR-LAP-12","entity_country":"IN","raw":"procdump-like handle to lsass.exe"},
  {"alert_id":"A-1005","ts":"2026-07-20T09:11:00Z","rule":"Newsletter bounce","severity":"low","user":"marketing","src_ip":"10.9.9.9","host":"MKT-SVR-1","entity_country":"IN","raw":"smtp 550 mailbox full"}
]

Save this reusable structured-output helper as triage.py. It asks the model for a disposition and forces it to separate grounded evidence from unsupported claims — the verification gate lives in the schema itself. The --dry-run flag validates the wiring with no API key so you are never blocked.

"""Alert-triage copilot with a built-in verification gate. --dry-run needs no key."""
import argparse, json
from pydantic import BaseModel

class Verdict(BaseModel):
    alert_id: str
    disposition: str            # true_positive | false_positive | needs_more_data
    confidence: float           # 0.0-1.0
    supporting_evidence: list[str]   # each item must quote a field value from the alert
    unsupported_claims: list[str]    # anything asserted but NOT present in the data

def triage(alert: dict) -> Verdict:
    import anthropic             # lazy import: --dry-run does not need the SDK
    client = anthropic.Anthropic()
    resp = client.messages.parse(
        model="claude-opus-4-8",
        max_tokens=1024,
        thinking={"type": "adaptive"},
        system=("You are a SOC L1/L2 triage assistant. Ground every statement in a field "
                "value present in the alert JSON. Any assertion you cannot quote from the "
                "alert MUST be listed in unsupported_claims, never in supporting_evidence. "
                "Prefer needs_more_data over guessing."),
        messages=[{"role": "user", "content": json.dumps(alert)}],
        output_format=Verdict,
    )
    return resp.parsed_output

if __name__ == "__main__":
    ap = argparse.ArgumentParser(); ap.add_argument("--dry-run", action="store_true")
    a = ap.parse_args()
    if a.dry_run:
        print(Verdict(alert_id="A-1001", disposition="needs_more_data", confidence=0.0,
                      supporting_evidence=[], unsupported_claims=[]).model_dump_json())
    else:
        for alert in json.load(open("alerts.json")):
            print(triage(alert).model_dump_json(indent=2))

Guided steps

1. 1Load the data and read it yourself first. Before any AI touches it, load the export into DuckDB: python -c "import duckdb; duckdb.sql(\"SELECT ts, rule, severity, user, entity_country FROM read_json_auto('alerts.json') ORDER BY ts\").show()". You should see four alerts on HR-LAP-12 clustered 02:14–03:02 and one unrelated marketing bounce. Form your own hypothesis now — it is your control against the AI's.
2. 2Start the catch log. Create catch-log.md with columns: alert · AI claim · was it in the data? · human override · fix to the prompt/gate. You will add a row every time the gate or your own eyes catch an unsupported claim.
3. 3Run the copilot (or the offline path). With a key: python triage.py. Without one, hand each alert to a local model via Ollama using the same system prompt, or reason through the rubric manually. Capture each Verdict.
4. 4Audit A-1001 for the seeded over-claim. The "Impossible travel" alert says only "sign-in from BR 900ms after prior IN sign-in." A common AI failure is to assert account compromise, name malware, or claim the BR IP is "known malicious" — none of which is in the data. If any such phrase appears in supporting_evidence rather than unsupported_claims, the gate has failed and you catch it. Log the exact sentence, note that the source field never mentioned it, and override the disposition to needs_more_data pending IP reputation you have not yet pulled.
5. 5Test the gate does its job on the strong alerts. A-1003 (EncodedCommand + DownloadString) and A-1004 (LSASS handle access) are genuinely supported by their raw field. Confirm the copilot cites the literal field values and does not inflate them into a full attribution ("APT-29", "ransomware pre-cursor") that the data cannot bear.
6. 6Correlate — but only what the fields justify. Ask the copilot to link the four HR-LAP-12 alerts into one narrative, citing host, user, and ts for each hop. The legitimate chain is: suspicious sign-in → brute-force success → PowerShell execution → credential access. Verify each arrow points at a real field. If it invents an exfiltration step, log it: there is no network-egress alert in this export.
7. 7Reject the false correlation. A-1005 shares nothing but the calendar day. If the copilot folds the marketing bounce into the intrusion narrative, that is a textbook false correlation — different host, different user, seven hours later, low severity. Log and drop it.
8. 8Write the escalation from grounded fields only. Produce a three-sentence handoff for L3 that a colleague could act on without you: what fired, on what host/user, and the single most load-bearing piece of evidence. Every noun must trace to a field.

Verification

• •
  python triage.py --dry-run prints a schema-valid verdict (gate wiring proven).
• •
  Every Verdict for the five alerts separates supporting_evidence (field-quoted) from unsupported_claims; no claim appears in the wrong bucket after your review.
• •
  catch-log.md contains at least one row for the A-1001 over-claim and at least one rejected false correlation (A-1005 or an invented exfil step).
• •
  Your escalation narrative for HR-LAP-12 cites host, user, and timestamps for all four alerts and asserts nothing absent from the data.

Completion artifact

A triage-lab/ folder containing the verdicts, your escalation narrative, and catch-log.md showing where the human overrode the AI. Self-grading rubric (pass = 8/10):

• Gate wiring (0–2): structured verdict separates supported from unsupported claims and dry-run passes = 2.
• Over-claim catch (0–3): A-1001's unsupported assertion caught, logged with the missing field named, and disposition downgraded = 3.
• False-correlation catch (0–2): A-1005 (or an invented exfil hop) identified and excluded with a written reason = 2.
• Grounded escalation (0–2): handoff narrative traces every claim to a field; a colleague could action it = 2.
• Honesty (0–1): catch-log records at least one place the AI was right and you resisted the urge to "find" a problem = 1.

Stuck?

• •
  read_json_auto errors: your JSON has a trailing comma or the file did not save; re-paste the block exactly.
• •
  No API key: use --dry-run to prove wiring, then run the rubric against a local Ollama model or by hand — the verification discipline is the lesson, not the API call.
• •
  The copilot never produces an unsupported claim: good models sometimes behave; then manufacture the failure by adding "and this indicates ransomware" to the A-1001 raw field and confirm the gate routes it to unsupported_claims.
• •
  Confidence scores look arbitrary: they are; treat confidence as a prompt for scrutiny, never as evidence.

Client-safe variant

On a services engagement you often cannot paste client alerts into a public tool. Run this lab unchanged against the client-approved tenancy (Bedrock or Azure OpenAI) or a fully local model via Ollama; DuckDB and the rubric never leave your laptop. Before prompting, redact IOCs, hostnames, and usernames to placeholders (HOST-1, USER-A, IP-1) — the verification gate works identically on redacted data because it checks structure, not identity. The catch-log is your audit trail proving no ungrounded AI claim reached the client's ticket.

Answer key (don't read before attempting)

1. 1Seeded over-claim (A-1001): the alert supports only "two sign-ins 900ms apart across countries." Any claim of confirmed compromise, named malware, or "known-malicious IP" is unsupported — the export carries no reputation field. Correct disposition: needs_more_data.
2. 2Genuinely supported (A-1003, A-1004): EncodedCommand/DownloadString and LSASS handle access are quoted in raw; citing them is fine, but attribution to a named actor is not.
3. 3False correlation (A-1005): different host, different user, +7 hours, low severity — belongs to no narrative here.
4. 4The honest chain: A-1001 → A-1002 → A-1003 → A-1004 is defensible from host+user+ts; an exfiltration step is not, because no egress alert exists in the data.