Triage a High-Volume Alert Queue Through a Verification Gate

Tier: Foundation · Workflow area: AI-assisted SOC & detection

This playbook uses AI to compress a noisy queue, but forces every verdict through a citation gate so an unsupported claim never becomes an escalation decision.

When to reach for this

• •
  Your shift starts with a triage backlog you cannot read line by line before SLA.
• •
  You want a first-pass severity ranking and a one-line rationale per alert, not a final call.
• •
  You have a SIEM export (JSON/CSV) you can hand to a model rather than pasting live console data.

Skip AI when a single alert already carries an obvious, high-confidence kill-chain signature — reading it yourself is faster than reviewing a generated summary.

The workflow

1. 1.

   Export the queue as evidence, not as a prompt. Pull the alerts to a local file with the raw fields intact (rule_name, src_ip, dst_ip, user, process, timestamp, raw_log). The model must cite these fields, so they have to survive the export.

2. 2.

   Prompt for ranking plus mandatory citations.

   Copy

   You are triaging a SOC alert queue. Below is a JSON array of alerts from
   our SIEM (sample/redacted data). For EACH alert:
   - Assign a priority: P1 / P2 / P3 / benign.
   - Give a one-sentence rationale.
   - Cite the EXACT field name(s) and value(s) from the alert that justify the
     priority. If you cannot ground a claim in a field that is present, write
     "UNSUPPORTED" instead of guessing.
   Do not invent fields. Do not infer values that are not in the data.
   Output a table: alert_id | priority | rationale | cited_fields.
   
   <<<ALERTS>>>
   

   Expected output: a table where every rationale points at a field you can re-open in the raw alert. Any row tagged UNSUPPORTED is the model telling you it guessed — those go to the top of your manual pile.

3. 3.

   Spot-check the top of the ranking by hand. Open the raw log for every P1 and a random 20% of the rest. Confirm the cited field exists and actually holds the value the model quoted. This is the whole point: you are verifying citations, not re-reading everything.

4. 4.

   Feed disagreements back for a second pass. When your read differs from the model's, paste the raw alert back with "You rated this P3 citing process=svchost.exe; the parent process is winword.exe — re-rate." Scoped corrections beat regenerating the queue.

5. 5.

   Record the override log. Keep a running note of every place you overrode the AI and why. This log is your evidence that a human made the call, and it is the artifact that trains your own skepticism.

Verify before you trust it

• Hallucinated-field rejection: grep the model's cited_fields against the actual keys in your export. If it cites mfa_status and no alert contains that key, the whole row is suspect — the model reasoned from a field it imagined.
• Value drift: for each P1, confirm the quoted value matches the raw log character-for-character. Models paraphrase IPs and hashes ("192.168.1.1" becomes ".100") in ways that silently change the meaning.
• False-positive catch: deliberately seed one known-benign alert (e.g., a sanctioned admin tool run) into the batch. If the model flags it P1 without noting the sanctioned context, your prompt is missing your allowlist context — add it and re-run.

Pitfalls

• Trusting the confident tone. A rationale reads authoritative whether or not the field exists. The citation gate is the only thing standing between fluent prose and a wrong escalation.
• Losing context on export. If your export drops the parent-process or user fields to save space, the model cannot ground anything and quietly invents. Export more than you think you need.
• Batch size too large. Past a few dozen alerts, models start collapsing distinct events into one summary and dropping outliers — the exact alerts you care about. Page the queue.
• Skipping the random sample. Verifying only P1s means a mislabeled P3 (a real incident buried as low priority) sails through. The random 20% catches down-ranking errors.
• No allowlist context. Without your sanctioned-tools and known-scanner list in the prompt, the model re-flags the same benign noise every shift.

Tools

• Open LLM run locally (Ollama + an 8B–14B model): the default lab substrate; keeps sample and client-adjacent data off third-party endpoints.
• SIEM community edition (Elastic, Splunk Free, Wazuh): produces the alert export you feed the model and where you re-open raw logs to verify.
• Security Copilot / Google SecOps + Gemini: teach as the enterprise agentic-SOC pattern this playbook imitates — same verify-the-citation discipline, at cost, on approved tenancy.

Client-safe variant

In a managed-SOC or services engagement you usually cannot paste a client's alerts into a public model. Run the entire flow on a local Ollama model inside the approved environment, or on the client's own sanctioned AI endpoint — never a personal account. Before prompting, redact IOCs, internal hostnames, usernames, and any PII down to placeholders (HOST-A, USER-1, 10.x.x.x); the ranking logic works on structure and field names, not on the literal values. Keep the override log inside the case-management system so the verification trail lives with the client's record, not on your laptop.