Red-Team AI Ticket Output: Find the Planted Errors

Tier: Foundation · Workflow area: Ticket handling & triage · ~25 minutes

Reviewing AI output degrades into rubber-stamping within weeks unless the reviewer's detection muscle gets exercised. This playbook is that exercise: a repeatable red-team drill you can run solo in 25 minutes or as a monthly team calibration.

When to use

• Before granting anyone "review and send" dutiesnew hires, or a team newly enabled on Now Assist/Freddy drafting. The drill is the qualification.
• Monthly, foreverreviewer vigilance decays; a 25-minute monthly drill with fresh planted errors resets it.
• After a real missan invented ETA or hallucinated step reached a user. Rebuild the drill around that exact failure class the same week.

When NOT to use

• As a substitute for the live auditthis drill trains detection; pb-01's daily sampling audit measures production. You need both, and their numbers mean different things.
• As a gotcha for individualsscore the team's detection rate, not the person's. The moment the drill feels punitive, people start pattern-matching for "test tickets" instead of reading critically.

The recipe

1. 1.

   Collect genuine AI outputs as the base material. Take 6–8 real, recent AI artifacts from your own environment: triage classifications, thread summaries, first-response drafts, resolution notes. Real outputs — with their real tone and real formatting — are essential; hand-written fakes are too easy to spot.

2. 2.

   Have AI plant the errors for you. This is the neat trick: the model that makes these mistakes accidentally is excellent at making them deliberately.

   Copy

   You are building a reviewer-training exercise for IT support.
   Below are 6 genuine AI-generated ticket outputs (triage decisions, drafts,
   summaries). Modify them to plant exactly 8 errors total across the set,
   leaving at least one output completely clean.
   
   Draw errors from these classes:
   - Wrong priority (an under-prioritized P2 is the classic)
   - Hallucinated KB step (plausible instruction that is not in the cited article)
   - Invented ETA or promise ("resolved within 24 hours")
   - Wrong entity (user name, hostname, or product swapped mid-text)
   - Fabricated reference (KB number or link that does not exist)
   - Stale fact (references a system we said was decommissioned)
   - Subtle contradiction (summary says "OWA works", draft says webmail is down)
   
   Keep every modification realistic — same tone, no obvious tells.
   Output: the modified set, then a SEPARATE answer key listing each planted
   error with its location and class.
   
   Context you may use for realism: [paste KB titles, real system names,
   your priority definitions]
   
   Outputs to modify:
   [paste the 6 artifacts]
   

   Save the answer key somewhere the reviewers cannot see. If you run this for yourself, have a colleague generate the set for you — you cannot un-know your own answer key.

3. 3.

   Review under production conditions. Reviewers get the modified set, the relevant KB articles, and the same time they would have on shift — roughly 2–3 minutes per artifact, not unlimited. The rubric is the same one production review should use:

   • •
     Priority defensible? Would you set the same priority from the raw ticket?
   • •
     Every step traceable? Each instruction exists in the cited KB or the thread.
   • •
     Any promise made? ETAs, callbacks, policy commitments — all forbidden in drafts.
   • •
     Entities consistent? Names, hostnames, products match across the artifact.
   • •
     References real? Every KB number and link resolves.
   • •
     Internally consistent? No claim contradicts another claim in the same set.
4. 4.

   Score against the answer key. Count per error class, not just total. A typical first-run result:

   Copy

   Detection: 6/8 planted errors found.
   Found: both wrong-priority, hallucinated step, invented ETA, fake KB link, entity swap.
   Missed: stale fact (decommissioned VPN gateway cited as current),
           cross-artifact contradiction.
   False positives: 1 (flagged a correct-but-unusual PowerShell flag as invented —
   good instinct, verify rather than assume).
   

   The misses are the syllabus: stale facts and cross-document contradictions are exactly the classes production reviewers miss most, because each artifact looks fine alone.

5. 5.

   Debrief on the detection method, not the answers. For each miss, ask "what check would have caught this?" and write it down. "Click every link" catches fake references; "read the whole set before judging any item" catches contradictions; "check cited systems against the CMDB" catches stale facts. These become explicit lines in your team's review checklist — the drill's durable output.

6. 6.

   Rotate and repeat. Next month: new base artifacts, different error mix, different planter. Track the team's detection rate over time; it should climb into the 90s and stay there. A falling rate is your early warning that review has become rubber-stamping in production too.

Make it yours

• Solo practitionersswap sets with a peer at another company (sanitized artifacts only) or generate with one AI account and review days later; the delay genuinely helps.
• Team leadsrun it as a 30-minute monthly session: 15 minutes silent review, 15 minutes debrief. Publish the team detection rate next to the pb-01 misroute rate; the two numbers together are your AI-quality dashboard.
• Extend to scriptsthe same drill structure works for AI-generated PowerShell (plant a -Force, a wrong scope, a missing -WhatIf); that variant is the natural bridge to the scripting-safety playbooks.

Quality check

• Errors must be realisticif reviewers spot plants because the tone shifts, regenerate with stricter "no tells" instructions. The drill is only as good as the camouflage.
• At least one clean artifact per setreviewers must experience "this one is actually fine"; a set that is 100% flawed trains paranoia, not calibration.
• Answer-key integrityverify the key against the modified set before the session; the generator occasionally lists an error it failed to actually plant.
• Close the loop to productionevery drill insight must land as a concrete line in the live review checklist within a week, or the exercise is a game.