Cluster False Positives and Find the Root Cause of Alert Noise

Tier: Core · Workflow area: AI-assisted SOC & detection

This playbook uses an LLM to cluster a mass of closed false positives, find the shared root cause behind each cluster, and propose a specific rule change — with the discipline that every proposed tuning is proven against real closed cases before it silences anything.

When to reach for this

• •
  A handful of rules generate most of your benign-closed volume and you want to fix the source, not keep clicking "close."
• •
  You have an export of recently closed alerts with their disposition and closing notes.
• •
  You are preparing a detection-tuning proposal and need the evidence to justify it.

Skip AI when a single rule is obviously misfiring — just fix it. This is for when the noise is spread across many rules and the pattern is not obvious.

The workflow

1. 1.

   Export closed false positives with their fields and dispositions. Pull a few weeks of benign-closed alerts including rule_name, the triggering fields, and the analyst close reason. The close notes are gold — they often already name the root cause in prose.

2. 2.

   Prompt for clustering and root-cause hypotheses.

   Copy

   Below is a JSON array of alerts we closed as false positive (sample data),
   each with the rule that fired and the analyst's close note.
   1. Cluster them by shared root cause (not just by rule name — a cause may
      span rules, and one rule may have several causes).
   2. For each cluster: name the likely root cause, list the alert_ids in it,
      and quote the evidence (fields/close-notes) that supports the cause.
   3. Propose ONE specific, narrow tuning per cluster (an exclusion, a threshold,
      a field condition). State exactly what it would suppress.
   Do not propose disabling a rule outright. Flag any cluster where the cause
   is ambiguous rather than guessing.
   

   Expected output: named clusters, each with member IDs, cited evidence, and a narrow tuning proposal — plus an "ambiguous" bucket you must investigate by hand.

3. 3.

   Validate each proposed exclusion against the full closed set. For every tuning, query how many of the closed false positives it would have suppressed. This is the safety test: a good exclusion suppresses the noise cluster and nothing else.

4. 4.

   Test the exclusion against known true positives. Run the same proposed condition against a set of confirmed real incidents. If it would have suppressed even one true positive, reject it — a quieter queue that hides real attacks is worse than a noisy one.

5. 5.

   Stage, don't silence. Deploy the exclusion in an audit/log-only mode first, watch it for a cycle, and confirm the suppressed volume matches the cluster and nothing new goes dark.

Verify before you trust it

• Cluster-membership check: open a sample of alerts the model placed in a cluster and confirm they actually share the claimed cause. Models cluster on surface similarity (same rule name) rather than real cause; verify the cause, not the label.
• Suppression-scope check: run the proposed exclusion as a query over closed FPs and count the hits. If it matches far more or far fewer than the cluster size, the condition is wrong.
• False-positive catch (inverted): the critical test is running the exclusion against confirmed true positives. Any hit means the tuning would blind you to real attacks — reject it outright.

Pitfalls

• Clustering on rule name instead of cause. The model may group everything one rule produced as "one cause" when the rule misfires for three different reasons. Insist causes can span and split rules.
• Over-broad exclusions. A proposal like "exclude all activity from the vuln-scanner subnet" is easy to write and dangerous — attackers pivot through trusted subnets. Keep exclusions narrow and field-specific.
• No true-positive regression. Tuning validated only against false positives is half-tested. The true-positive check is what separates safe tuning from self-inflicted blindness.
• Silencing instead of fixing. Sometimes the root cause is a misconfigured log source, not a bad rule. An exclusion hides that; fixing the source removes the noise for good.

Tools

• SIEM community edition (Elastic, Splunk Free, Wazuh): produces the closed-alert export and is where you re-run exclusions against both FP and TP sets.
• Open LLM run locally (Ollama): does the clustering and root-cause reasoning without sending closed-case data off-site.
• MITRE ATT&CK Navigator: cross-check that a proposed exclusion does not blind you to a technique you are supposed to cover.

Client-safe variant

Closed-case data reveals a client's detection posture and their real incidents, so it stays inside the boundary. Run the clustering on a local Ollama model or the client's sanctioned endpoint. Redact hostnames, users, and internal IPs to placeholders before prompting — clustering works on the pattern of fields and close-notes, not the literal identifiers. Crucially, keep the true-positive regression set inside the client environment and never let a proposed exclusion leave log-only mode without the client's detection owner signing off; you are proposing to create blind spots in their coverage, and that decision is theirs to approve.