This module installs the detection engineer's evidence bar: an AI-authored rule is only a claim until test data shows it firing on the attack, staying silent on benign telemetry, and surviving conversion into the SIEM that will actually run it.
AI can draft a Sigma rule from a described behavior, compile it to KQL, SPL, and Elastic in seconds, and turn a threat report into a candidate detection before your coffee cools. This module teaches the discipline that makes that speed trustworthy: an AI-authored rule is a claim that it detects the behavior, and a claim is worthless until test data proves it fires on the bad and stays silent on the good.
Ananya is a detection engineer for a fintech's internal SOC, Elastic and Splunk in production, pySigma in her pipeline. A CTI report lands describing a fresh commodity-loader campaign: it establishes persistence by registering a scheduled task that runs a script from %APPDATA%. Her job by end of day: a working detection across both SIEMs.
She asks her AI assistant to author a Sigma rule for the behavior. Eight seconds later it hands her clean, schema-valid YAML: a process_creation rule matching schtasks.exe with /create. It compiles through pySigma to Elastic and SPL without a complaint. It looks done. Old-Ananya, on a deadline, would ship it — it parses, it converts, the deadline is met.
Instead she does the one thing that separates a detection engineer from a rule generator: she tests it against data. She runs Atomic Red Team test T1053.005 in her lab to generate a true-positive event — the actual malicious scheduled-task creation. The rule fires. Good.
Then she replays a day of benign endpoint telemetry from a build server. The rule fires 1,400 times. Every CI job that schedules a task, every legitimate admin script, every backup registration — all schtasks /create, all matched. As written, the rule would bury a real detection under 1,400 false positives a day and get itself disabled within a week.
She feeds the failure back: "This fires on all scheduled-task creation. The malicious behavior specifically runs from %APPDATA% or %TEMP%. Constrain to that, and show me the test events that should now pass and fail." The refined rule adds a CommandLine condition for the user-writable paths.
The Atomic true positive still fires; the 1,400 benign events drop to two — both genuinely odd admin scripts running from a temp path, worth a look. That is a detection. The first version was a liability wearing valid syntax.
She converts the tested rule with pySigma to Elastic Query DSL and SPL, then does her second verification: she confirms the converted rules fire on the same test events in the actual SIEMs, because a Sigma field that maps cleanly on paper can land on a differently-named or unpopulated field in a specific data source.
The Elastic version matches. The SPL version returns nothing — the field mapping pointed at Process_Command_Line where her Splunk data uses process. She fixes the mapping, re-tests, and both fire.
The lesson in one line. The AI wrote three syntactically perfect rules in under a minute, and all three would have failed silently in production — one buried in false positives, one pointed at an empty field. What made them real detections was not the authoring, which AI did brilliantly, but the testing, which only Ananya could insist on.
Authoring is minutes; the value is in the test data. Time the AI saved on syntax: enormous. Time Ananya spent proving the rules actually work: forty minutes, and non-negotiable.
Here is a described behavior turned into a Sigma rule, the way M2 wants it done — with the over-broad first draft shown honestly:
# AI-drafted from: "malware persists via scheduled task running from APPDATA"
# FIRST DRAFT — over-broad, will bury you
title: Scheduled Task Creation
logsource: {category: process_creation, product: windows}
detection:
sel:
Image|endswith: '\schtasks.exe'
CommandLine|contains: '/create'
condition: sel
# REFINED after testing against benign build-server telemetry
title: Scheduled Task Persistence From User-Writable Path
logsource: {category: process_creation, product: windows}
detection:
sel:
Image|endswith: '\schtasks.exe'
CommandLine|contains|all: ['/create', '/tr']
paths:
CommandLine|contains:
- '\AppData\'
- '\Temp\'
- '%APPDATA%'
- '%TEMP%'
condition: sel and paths
falsepositives: [Admin scripts legitimately staged in temp paths]
level: high
The difference between the two is the entire module. The first encodes the tool (schtasks); the second encodes the malicious specificity (persistence from a path attackers use and clean software rarely does). AI reaches for the first because the CTI report names the tool; only test data reveals that the tool alone matches a thousand benign events.
The rule you enforce on every AI-authored detection: generation is a hypothesis, test data is the proof — a rule that has not fired on a true positive and stayed silent on a corpus of benign data is not a detection, it is a guess in YAML. The verification loop is fixed and cheap.
| Test | Source | Pass criterion |
|---|---|---|
| True positive fires | Atomic Red Team / recorded attack telemetry | Rule matches the malicious event |
| Benign silence | A day of your real production telemetry | Near-zero matches; every match explainable |
| Specificity | Compare match count on benign corpus | Fewer, and each one a real edge case |
| Converted rule fires | Same test events in the target SIEM | Match survives field mapping, not just Sigma |
A syntactically perfect, cleanly-converting AI-authored rule can still detect nothing or bury you in noise — validity is not efficacy, and only test data against both true-positive and benign corpora tells the two apart.
The generation prompt should ask for the behavior and the false-positive surface in the same breath:
Author a Sigma rule for this behavior. Constrain it to what is SPECIFIC to
the malicious activity, not the tool it uses. List the benign activities
that would also match a naive version, and add conditions that exclude them.
Output the rule, its expected true positives, and its likely false positives.
Conversion is where AI and deterministic tooling divide labor cleanly. The safe pattern uses pySigma's backends to do the actual translation and reserves the LLM for the judgment pySigma cannot make — field mapping to your data model:
# Deterministic conversion — pySigma backends, not the LLM, produce the query
sigma convert -t elasticsearch -p ecs_windows persist_apdata.yml
sigma convert -t splunk -p splunk_windows persist_apdata.yml
sigma convert -t kusto -p sentinel_windows persist_apdata.yml
Prefer pySigma's tested backends over an LLM free-hand "translate this to SPL" whenever a backend exists — the compiler produces correct, deterministic syntax, and it does not hallucinate an operator that does not exist. Use the LLM for the part the compiler genuinely cannot do: choosing the right field-mapping pipeline when your data source names fields differently, and explaining a conversion that landed on an empty field.
Ananya's SPL failure — right rule, wrong field name — is the canonical conversion trap, and no amount of syntactic validity catches it. The only guard is re-testing the converted rule in the target SIEM against the same true-positive event.
| Conversion task | Do it with | Why |
|---|---|---|
| Sigma → KQL/SPL/Elastic/YARA-L syntax | pySigma backends | Deterministic, tested, no hallucinated operators |
| Choosing the field-mapping pipeline | LLM + your knowledge of the data | Requires knowing how your source names fields |
| Diagnosing an empty converted rule | LLM over the schema | Explains the mapping gap; you confirm |
| Proving the converted rule works | Test data in the target SIEM | Field mapping only proven by a real match |
Let pySigma compile the syntax and reserve the LLM for field-mapping judgment — then re-test the converted rule in the actual SIEM, because a mapping that is correct on paper can land on an empty field in production.
A threat report is unstructured prose; a detection pipeline turns it into a tested rule through four stages, and structured output keeps each stage honest. First, extraction of IOCs and ATT&CK techniques to a schema:
import anthropic
from pydantic import BaseModel
class Technique(BaseModel):
attack_id: str # e.g. T1053.005
behavior: str # what the report describes, verbatim-grounded
source_quote: str # the sentence in the report this came from
class Extraction(BaseModel):
iocs: list[str] # hashes, domains, IPs as written in the report
techniques: list[Technique]
detection_candidates: list[str] # behaviors specific enough to detect
client = anthropic.Anthropic()
response = client.messages.parse(
model="claude-opus-4-8",
max_tokens=3000,
thinking={"type": "adaptive"},
messages=[{
"role": "user",
"content": f"""Extract detection-relevant intel from this threat report.
<report>{cti_report_text}</report>
Map behaviors to MITRE ATT&CK technique IDs. Every technique must quote the
report sentence it came from. Do NOT invent technique IDs or IOCs not present
in the text. Flag behaviors too generic to detect specifically.""",
}],
output_format=Extraction,
)
extraction = response.parsed_output
The grounding instruction matters more here than anywhere: an LLM asked to map prose to ATT&CK will confidently emit real-looking technique IDs that the report never supports, and a wrong T-number propagates into your coverage metrics as a technique you falsely believe you cover. Forcing a source quote per technique makes every mapping checkable against the sentence that justifies it.
For IOC extraction destined for a threat-intel platform, emit STIX rather than a loose list — the same structured-output discipline, typed to the STIX schema, so the bundle drops straight into MISP or OpenCTI. The verification here is different from behavior mapping and just as important: an IOC extracted with the wrong type is a silent defect. A model that reads a hostname as an IP, or a file path as a domain, produces a STIX object that validates cleanly and then never matches, because you are hunting the right value in the wrong field. Spot-check a sample of extracted indicators against their source lines before the bundle leaves your hands.
The extracted behaviors then flow into the authoring and testing loop above: candidate rule, true-positive test, benign silence, cross-SIEM conversion, re-test. Nothing about the CTI origin exempts a rule from the test corpus — a detection derived from a threat report is exactly as unproven as one you invented, and the report's confident prose about "attacker behavior" is not evidence your rule fires on it.
A hallucinated ATT&CK technique ID does not throw an error — it silently inflates your coverage map. Every AI-generated mapping must quote the report sentence that supports it, and every extracted IOC must be checked for the right type before it reaches your TIP.
Two maintenance tasks keep a growing ruleset from rotting, and both are AI-assisted analysis you verify.
Coverage gap analysis. Export your rule inventory mapped to ATT&CK and ask the model to reason over the matrix: "Given these 180 deployed rules and their technique mappings, which techniques in the Persistence and Defense Evasion tactics have zero coverage, and which have only brittle string-based coverage that a minor variation would evade?" The output overlays onto MITRE ATT&CK Navigator as a gap map. The second half of that question — brittle coverage — is where the AI earns its keep, because a rule that keys on a literal command string looks like coverage on the Navigator heatmap and evaporates the moment an attacker renames a binary or reorders an argument. The AI reads your rules for that fragility faster than you can.
The verification is non-negotiable: the AI's "no coverage" claim is a hypothesis, and so is its "brittle coverage" claim. Confirm a genuinely uncovered technique by running its Atomic test and watching nothing fire, exactly as you would confirm a real detection by watching something fire — and confirm a "brittle" finding by generating a lightly-varied version of the attack and watching the supposedly-covering rule miss it. A gap map you have not tested is a second confident story, and this whole module is about not shipping those.
Regression and duplication. As rules accumulate they drift and overlap — two rules for the same behavior authored six months apart, a rule broadened last quarter that now silently shadows a narrower, higher-fidelity one. Ask the model to cluster the ruleset by behavior and flag redundancy and drift: "Which of these rules match overlapping event sets? Which appear to have been widened over time in ways that would now swallow a more specific rule's alerts?" Then re-run your test corpus to confirm that consolidating duplicates does not lose a true positive — the AI proposes the merge, the test data proves it safe. The failure to guard against is a consolidation that looks tidy and quietly drops the one narrow rule that caught a real technique, which only the true-positive corpus will reveal.
| Maintenance task | AI proposes | You prove with |
|---|---|---|
| Coverage gap vs ATT&CK | Uncovered / brittle techniques | Atomic test shows nothing fires |
| Duplicate / shadowed rules | Redundancy clusters | Test corpus after merge keeps all true positives |
| Drift (broadened rules) | Rules matching more than intended | Benign corpus match-count trend |
Coverage claims and consolidation proposals are hypotheses like any detection — a "gap" is proven when the Atomic test fires nothing, and a "safe merge" is proven when the true-positive corpus still passes after it.
Treat "we have no coverage for this technique" the same way you treat a detection — as a claim to test. Run the Atomic test and confirm nothing fires before you write it up as a gap.
Everything above assumes your detections live in version control, tested in CI, reviewed by a peer before they deploy — and AI raises the value of that discipline rather than lowering it. When anyone can generate a plausible rule in eight seconds, the pull request review becomes the load-bearing quality gate, and the reviewer's job shifts from "is the syntax right" (pySigma and CI answer that) to "does the test evidence prove this rule works."
The review checklist for an AI-authored detection is short and specific. Is there a true-positive test artifact showing it fires on the behavior? Is there a benign-corpus result showing an acceptable false-positive rate? For a converted rule, is there evidence it fires in the target SIEM, not just in Sigma? Does the rule encode the malicious specificity or just the tool? A PR that generates a rule but attaches no test evidence should not merge, no matter how clean the YAML — the absence of evidence is the finding.
| PR review question | Merge if | Block if |
|---|---|---|
| True-positive evidence attached? | Test event and a matching fire are shown | Only the rule, no proof it detects |
| Benign false-positive rate | Measured on real telemetry, acceptable | Untested, or "looks fine" |
| Converted-rule proof | Fires in the target SIEM | Only Sigma-side validation |
| Specificity | Keys on malicious behavior | Keys on a benign-shared tool or string |
This is where AI-assisted detection engineering compounds safely: the model does the authoring and conversion at machine speed, CI runs the deterministic checks, and the human review enforces the one thing neither can fake — that test data, not fluent YAML, is what earned the rule its place in production. The generation getting faster is precisely why the evidence bar at the review gate must not move.
For services and multi-tenant SOCs, the same India-aware constraint from M1 applies to detection content. A CTI report about a client's environment, or telemetry samples used as test data, can carry PII and client-confidential detail that must not be pasted into a public model. Use client-approved or local endpoints for anything tenant-specific, keep generic Sigma authoring (which needs no client data) on whatever tool is approved, and redact sample events down to the fields the rule actually keys on before prompting. A detection rule authored from a redacted behavior is portable and shareable; the raw telemetry it was tested against is not.
This split is a genuine advantage for detection work specifically. The behavior a rule detects — "scheduled task persisting from a user-writable path" — is generic threat knowledge with no client data in it, so authoring can happen on any approved tool. Only the testing touches real telemetry, and that stays inside the client-approved boundary on local models or enterprise endpoints. Structuring your pipeline along that seam — generic authoring outside, tenant-specific testing inside — lets a services team get the full speed of AI-assisted authoring without ever putting a client's logs where they must not go.
This module anchors LAB-03 (Sigma authoring + cross-SIEM conversion) and LAB-04 (CTI-report → detection pipeline): from a described behavior, generate a Sigma rule, compile it to KQL, SPL, and Elastic via pySigma, and validate against test events; then feed an unstructured threat report, extract ATT&CK TTPs and IOCs to STIX, emit a candidate detection, and test it. The artifacts — one Sigma rule with three passing backend conversions, and a mapped-TTP table with a tested rule and its true-positive/benign-silence evidence — are the strongest interview proof this module produces: anyone can generate a rule; you can show it firing on the attack, silent on the noise, and working after conversion.
Your first 15 minutes: before you ship any AI-authored detection this week, replay it once against a day of benign telemetry — if it fires more than a handful of times, it is a liability wearing valid syntax, not a rule, and it goes back for a constraint.
Check yourself: Your AI-authored Sigma rule compiles, converts cleanly to KQL and SPL, and passes schema validation. Why is none of that evidence that it is a working detection — and what two tests make it one?
Companion playbooks — take these into your week alongside LAB-03 and LAB-04: pb-06 (Author a Sigma Detection from a Described Threat Behavior), pb-07 (Convert One Sigma Rule to KQL, SPL, and Elastic with Validation), and pb-09 (Turn a CTI Report into a Tested Detection with ATT&CK Mapping).