Generate Detection Test Data and Regression-Check Rules for Drift

Tier: Advanced · Workflow area: Detection engineering with AI

This playbook uses AI for two linked jobs — generating known-good and known-bad test events for a rule, and running a regression pass across the rule set to find drift, duplication, and dead detections — with the hard rule that generated test data is validated against ground truth before it certifies anything.

When to reach for this

• •
  A new or changed detection needs positive and negative test cases and you lack a real sample.
• •
  Your rule set has grown large enough that overlaps and stale rules are likely but not visible.
• •
  You are standing up a detection CI pipeline and need repeatable test fixtures.

Skip generation when you already have a real, labeled malicious sample — real data beats synthetic every time. Use AI to fill gaps, not to replace ground truth.

The workflow

1. 1.

   Generate test events matched to the rule's fields. Give the model the detection and ask for events that exercise the exact condition — both the case that must trigger and the near-miss that must not.

   Copy

   For this Sigma rule (pasted below), generate synthetic Sysmon EID 10 test
   events (sample data, JSON):
   - 3 TRUE POSITIVE events that should match the condition, varied so they do
     not all look identical.
   - 3 TRUE NEGATIVE events that are close but must NOT match (e.g. sanctioned
     LSASS access, different GrantedAccess mask).
   Use only real EID 10 fields. For each event, state in a comment WHY it should
   or should not match, referencing the rule's condition.
   

   Expected output: labeled positive/negative events, each annotated with why it lands on its side of the boundary. The annotations are what you check.

2. 2.

   Validate the test data against the rule before trusting it. Run the compiled rule over the generated events. Every "true positive" event must match; every "true negative" must not. If a generated positive does not fire, either the event or your understanding of the rule is wrong — resolve it before this data certifies anything.

3. 3.

   Cross-check near-misses against a real known-bad. Synthetic data can be too clean — it exercises the rule as written, not as attackers behave. Where you have even one real malicious sample (Atomic Red Team artifact), confirm the rule fires on it too, so you are not only testing the synthetic ideal.

4. 4.

   Run the regression pass across the rule set. Ask the model to compare rules for overlap and drift, grounded in the rule bodies.

   Copy

   Here are N Sigma rules (pasted). Identify:
   - Duplicate/near-duplicate rules (same behavior, cite the overlapping
     conditions).
   - Rules that are strict subsets of another (redundant coverage).
   - Rules referencing fields/log sources we no longer collect (drift — I will
     confirm against our schema).
   For each finding cite the specific conditions. Do not merge or delete
   anything; propose, and mark low-confidence findings.
   

5. 5.

   Confirm every regression finding by hand. Overlap and subset claims are hypotheses. Open the cited conditions and verify before you retire or merge any rule — a "duplicate" may differ in one field that matters.

Verify before you trust it

• Ground-truth check: run generated positives and negatives through the actual compiled rule. A generated "positive" that does not fire is a hallucinated test case — discard it, do not adjust the rule to match it.
• Realism check: compare a synthetic malicious event to a real one. If the synthetic version only matches because it is unnaturally clean, the test gives false confidence.
• False-positive catch on regression: before retiring a "redundant" rule, confirm it is truly a subset by running both against known-bad data. If the "redundant" rule catches something the other misses, it is not redundant — keep it.

Pitfalls

• Certifying on synthetic data alone. The core risk: green tests against events the model wrote to pass. Anchor to real known-bad.
• Over-clean negatives. Generated "benign" events often lack the messiness of real logs, so the rule looks precise when it would flood on production noise.
• Trusting duplicate claims. Two rules that look identical may differ in a GrantedAccess mask that changes what they catch. Verify overlaps against data, not just text.
• Retiring drifted rules too fast. A rule referencing a field you "no longer collect" may protect a niche source you forgot about. Confirm the source is truly gone.

Tools

• Atomic Red Team: the source of real, safe known-bad artifacts to anchor synthetic tests against ground truth.
• Sigma + pySigma: compiles rules so generated events can be run through them for validation and regression checks.
• Open LLM run locally (Ollama): generates test events and reasons over the rule set offline.

Client-safe variant

Test-data generation is naturally client-safe because it produces synthetic events — but the rules you feed the model reveal the client's detection coverage. Run both the generation and the regression pass on a local Ollama model or the client's approved endpoint. Keep the rule set and all validation inside the client's environment; generated test events are synthetic and portable, but the real known-bad anchors and the retire/merge decisions belong to the client's detection owner. Ship the labeled test fixtures into the client's detection CI so every future rule change re-runs the same proven positive/negative cases.