Audit an AI Triage Rationale to Catch Its Wrong Conclusions

Tier: Core · Workflow area: AI-assisted SOC & detection

You are not asking the model to be right; you are building a reflex for catching when it is wrong.

When to reach for this

• •
  Your SOC has an AI (agentic or prompt-based) producing triage verdicts and rationales you must sign off on.
• •
  You want a repeatable check that scales beyond your own memory of every failure mode.
• •
  You are calibrating trust in a new AI triage tool before you let it touch real severity decisions.

Do this on every AI verdict that would trigger an escalation or a close-as-benign — those are the two directions a wrong conclusion hurts most.

The workflow

1. 1.

   Separate the verdict from its evidence. Take the AI's output — verdict, rationale, cited fields — and the raw alert it was drawn from. Keep them as two distinct blocks so the audit compares them rather than re-reading the summary.

2. 2.

   Run an adversarial self-check prompt. Use a different model instance (or a fresh context) as the auditor, so it does not defend the original reasoning.

   Copy

   You are auditing another AI's triage verdict against the raw evidence.
   Do not decide the alert yourself. Instead, for each claim in the rationale:
   - Quote the claim.
   - State whether the raw evidence SUPPORTS, CONTRADICTS, or does NOT MENTION it.
   - If CONTRADICTS or NOT MENTIONED, explain the gap.
   Then answer: does the verdict survive if we drop every unsupported claim?
   
   [VERDICT + RATIONALE]
   ...
   [RAW ALERT EVIDENCE]
   ...
   

   Expected output: a claim-by-claim ledger. The killer line is the last one — if the verdict collapses once unsupported claims are removed, it was resting on invented evidence.

3. 3.

   For structured pipelines, enforce the audit in code. When the triage AI is part of an app, parse the audit into a schema so an unsupported verdict cannot auto-close a case.

   Copy

   import anthropic
   from pydantic import BaseModel
   
   class ClaimCheck(BaseModel):
       claim: str
       status: str  # SUPPORTED | CONTRADICTS | NOT_MENTIONED
       gap: str
   
   class Audit(BaseModel):
       claims: list[ClaimCheck]
       verdict_survives: bool
       reasoning: str
   
   client = anthropic.Anthropic()
   response = client.messages.parse(
       model="claude-opus-4-8",
       max_tokens=2000,
       thinking={"type": "adaptive"},
       messages=[{"role": "user", "content": audit_prompt}],
       output_format=Audit,
   )
   result = response.parsed_output
   if not result.verdict_survives:
       route_to_human(result)  # never auto-close on unsupported evidence
   

4. 4.

   Escalate every collapse to a human. Any verdict where verdict_survives is false, or where the auditor found a CONTRADICTS, goes to manual review. That is the whole safety property.

5. 5.

   Log the catches. Keep a tally of what the audit caught. Recurring failure patterns (the AI always over-trusts svchost.exe) become new rules in your triage prompt.

Verify before you trust it

• The auditor can hallucinate too: spot-check the auditor's SUPPORTED/CONTRADICTS labels against the raw evidence yourself. An audit you never verify is just a second unverified opinion.
• Citation check: confirm each "SUPPORTED" claim really maps to a field present in the raw alert, not to the auditor agreeing with fluent prose.
• False-positive catch: feed the audit a verdict you know is wrong (a benign admin action labeled P1) and confirm the audit flags the contradiction. If it rubber-stamps, your audit prompt is too deferential — sharpen it.

Pitfalls

• Same-context auditing. Asking the same conversation to check itself yields self-agreement. Use a fresh context or a different model as the auditor.
• Auditing the summary, not the evidence. If you only give the auditor the rationale, it grades prose quality, not truth. It must see the raw alert.
• Treating "verdict_survives: true" as proof. It means the stated evidence supports the stated verdict — not that the evidence is complete. A verdict can survive the audit and still miss context nobody provided.
• Never verifying the auditor. Two AIs agreeing is not verification. A human still spot-checks.

Tools

• Any LLM plus an explicit eval rubric: the audit is a rubric applied by a second model; the rubric is the durable asset, not the model.
• Open LLM run locally (Ollama): runs both the triage and the auditor offline for sensitive queues.
• Structured-output SDK (Claude messages.parse): turns the audit into a machine-checkable gate that blocks auto-close on unsupported verdicts.

Client-safe variant

The audit itself is a safety control, so it belongs inside the client boundary. Run both the original triage model and the auditor on a local Ollama model or the client's approved AI endpoint, never a personal account. Redact the raw alert evidence — hostnames, users, internal IPs to placeholders — before either model sees it; the audit checks structural support ("is this field present?"), which survives redaction intact. Store the claim-by-claim ledger and the catch log in the client's case system so the verification trail is auditable by the client, not held on your machine.