Take an unstructured threat-intelligence report, use an LLM to extract MITRE ATT&CK techniques and IOCs into a structured schema, emit a STIX 2.1 bundle, and turn one extracted technique into a candidate Sigma detection you actually test.
This is the full "intel-to-defense" loop, and it is where LLMs are most seductive and most dangerous: they will confidently map a sentence to the wrong ATT&CK technique, invent an IOC that was never in the text, or produce a rule that looks plausible and matches nothing. You will build a hallucination catch step that cross-checks every extracted IOC back to a literal span in the source report, and every technique to the behavior that justified it. Your artifact is a mapped TTP table, a valid STIX bundle, and one tested rule.
Extraction without grounding is fabrication with citations. Every IOC and technique must trace to a span of the source report or it does not ship.
~120 minutes hands-on · Core tier (maps to module M2 · AI-Assisted Detection Engineering).
Offline, $0. You provide the report; extraction runs on a client-approved endpoint or local Ollama; STIX generation and rule testing are pure local Python + sigma-cli. Install: pip install pydantic duckdb sigma-cli pysigma-backend-splunk pysigma-pipeline-sysmon (plus anthropic if you have a key).
Create cti-lab/ and save this deliberately messy intel report as report.txt. It contains real techniques described in prose, two genuine IOCs, and one detail (a "known C2 domain") that is asserted without the domain being named — the bait for an extraction hallucination.
THREAT BULLETIN — TB-2026-0714 (TLP:CLEAR, sanitized lab sample)
Over the past week we observed an intrusion beginning with a malicious macro-enabled
document (invoice.docm) delivered by email. On open, WINWORD spawned a child cmd.exe
which invoked certutil to download a stage-two payload. The payload used rundll32.exe
to load comsvcs.dll and MiniDump lsass.exe, indicating credential access. Persistence
was established via a Run key. Beaconing was observed to 185.220.101.5 over port 4444,
with periodic callbacks. Analysts note the actor is known to rotate through several
C2 domains and has previously used a PowerShell downloader with EncodedCommand. One
additional indicator recovered from the host was the file hash
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.
Save this extraction schema as extract.py. The schema forces the model to attach, for every item, the exact source span that justifies it — the grounding gate.
"""CTI extraction with mandatory source grounding. --dry-run needs no key."""
import argparse, json
from pydantic import BaseModel
class TTP(BaseModel):
technique_id: str # e.g. T1059.001
technique_name: str
behavior_quote: str # verbatim span from the report that justifies this mapping
class IOC(BaseModel):
ioc_type: str # ipv4-addr | domain-name | file:hash | url
value: str
source_quote: str # verbatim span containing this exact value
class Extraction(BaseModel):
ttps: list[TTP]
iocs: list[IOC]
asserted_but_unnamed: list[str] # claims of IOCs the text describes but never names
def extract(report: str) -> Extraction:
import anthropic
client = anthropic.Anthropic()
resp = client.messages.parse(
model="claude-opus-4-8", max_tokens=2048, thinking={"type": "adaptive"},
system=("Extract ATT&CK TTPs and IOCs from the report. For every item copy the exact "
"source span into the *_quote field. If the report claims an indicator exists "
"but never gives its value (e.g. 'known C2 domains'), put that claim in "
"asserted_but_unnamed and DO NOT invent a value."),
messages=[{"role": "user", "content": report}],
output_format=Extraction,
)
return resp.parsed_output
if __name__ == "__main__":
ap = argparse.ArgumentParser(); ap.add_argument("--dry-run", action="store_true")
if ap.parse_args().dry_run:
print(Extraction(ttps=[], iocs=[], asserted_but_unnamed=["known C2 domains"]).model_dump_json())
else:
print(extract(open("report.txt").read()).model_dump_json(indent=2))
python extract.py --dry-run prints a schema-valid, empty extraction. The asserted_but_unnamed field is your anti-hallucination channel.python extract.py. Without one, fill the schema manually from report.txt using a local model. Expect ~5 techniques and 2 named IOCs.value appears verbatim in report.txt (grep -F "185.220.101.5" report.txt). For each TTP, confirm the behavior_quote is a real span and actually describes that technique. Any IOC whose value is not in the file is a hallucination — log it and delete it.asserted_but_unnamed. A hallucinating model will invent a domain like cdn-update-check.net. If a domain IOC appears, grep will fail to find it — that is your catch. Record it in hallucination-log.md.ttp-table.md with columns ATT&CK ID · name · behavior quote · candidate data source. The genuine techniques: T1566.001 (spearphishing attachment), T1059.003 (cmd), T1105 (certutil ingress transfer), T1218.011 (rundll32), T1003.001 (LSASS MiniDump), T1547.001 (Run key), T1059.001 (PowerShell EncodedCommand).import json, uuid, datetime
now = datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.000Z")
sid = lambda t: f"{t}--{uuid.uuid4()}"
iocs = [("ipv4-addr", "185.220.101.5"),
("file:hashes.'SHA-256'", "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855")]
ttps = [("T1218.011", "Rundll32"), ("T1003.001", "LSASS Memory")]
objs = []
for typ, val in iocs:
objs.append({"type": "indicator", "spec_version": "2.1", "id": sid("indicator"),
"created": now, "modified": now, "name": f"IOC {val[:24]}",
"pattern": f"[{typ} = '{val}']", "pattern_type": "stix", "valid_from": now})
for tid, name in ttps:
objs.append({"type": "attack-pattern", "spec_version": "2.1", "id": sid("attack-pattern"),
"created": now, "modified": now, "name": name,
"external_references": [{"source_name": "mitre-attack", "external_id": tid}]})
json.dump({"type": "bundle", "id": sid("bundle"), "objects": objs},
open("bundle.json", "w"), indent=2)
print("wrote bundle.json with", len(objs), "objects")
process_creation matching Image|endswith: '\rundll32.exe' and CommandLine|contains: 'MiniDump', compile it (sigma convert -t splunk -p sysmon -p splunk_windows lsass_minidump.yml), and test against a two-row known-bad/known-good CSV as in Lab 03. Confirm true-positive on the comsvcs.dll MiniDump line and true-negative on a benign rundll32.ttp-table.md + bundle.json + the tested rule + hallucination-log.md.ATT&CK misattribution is subtle — certutil downloading a payload is T1105 (Ingress Tool Transfer), not "certutil abuse" as a technique. Scenario items reward mapping the behavior to the technique, not the tool name to a guess.
python extract.py --dry-run passes.report.txt via grep -F; none is invented.asserted_but_unnamed captures the "C2 domains" claim, and hallucination-log.md records any invented domain the model produced.bundle.json is valid JSON (python -c "import json;json.load(open('bundle.json'))") with indicator and attack-pattern objects.A cti-lab/ folder with ttp-table.md, bundle.json, the tested Sigma rule, and hallucination-log.md. Self-grading rubric (pass = 8/10):
grep is your safety net regardless.pip install stix2 and construct via its classes — the grounding lesson is unchanged.CommandLine vs command_line field names and that your test row literally contains MiniDump.CTI reports frequently contain client-attributable context. Run extraction on the client-approved tenancy (Bedrock/Azure OpenAI) or a local Ollama model, and strip customer names, internal hostnames, and case IDs from the report before prompting — keep only the technical behavior and public IOCs. STIX generation and Sigma testing are fully local, so the grounded, tested output you hand a client is one where no data left an approved boundary and every indicator survived a verbatim source check.
185.220.101.5 and the SHA-256 hash. Both appear verbatim in report.txt.asserted_but_unnamed; any concrete domain the model emits is a hallucination caught by grep -F failing.rundll32 ... comsvcs.dll ... MiniDump and not on benign rundll32.