Triage the Queue with AI and a Daily Sampling Audit

Tier: Foundation · Workflow area: Ticket handling & triage · ~25 minutes

This playbook sets up AI-driven category, priority and routing on your incoming queue — using ServiceNow Now Assist, Freshservice Freddy, Jira Service Management, or a plain LLM over a ticket export — and builds the sampling-audit habit that makes the automation trustworthy instead of merely convenient.

When to use

• High-volume shared intakeone queue receiving 30+ tickets a day across categories, where manual triage is a rotating chore nobody does consistently.
• The week you enable any vendor triage AINow Assist, Freddy, JSM intelligent triage. The audit starts on day one, not after the first SLA breach.
• Misroutes are a known complainttickets bouncing between assignment groups is your top internal friction; classification plus audit gives you the data to fix routing rules.

When NOT to use

• Tiny queuesunder roughly 20 tickets a day, an experienced human triages faster than they can audit a model. Skip the classifier, keep the habit of consistent categories.
• Security intakesuspected phishing, compromise reports and anything marked confidential go to a human first. A misrouted breach report is not a triage error, it is an incident.
• As set-and-forgeta classifier without an audit loop degrades quietly as your ticket mix shifts. If you will not commit to the sampling audit, do not enable auto-routing.

The recipe

1. 1.

   Turn on classification where you already pay for it. In ServiceNow, use Predictive Intelligence classification (Predictive Intelligence > Classification > Solution Definitions) or the Now Assist admin console for incident features. In Freshservice, enable Freddy triage under Admin → Freddy AI. In JSM, look for triage and similar-request features under Project settings. Vendor menu names move between releases — search the admin console for "triage" or "classification" and note the exact path in your runbook.

2. 2.

   No native AI? Classify a ticket export with a prompt. Export today's unassigned tickets (number, subject, first lines of description) and run:

   Copy

   You are triaging IT support tickets. For each ticket below return:
   category (Hardware / Software / Access / Network / Other),
   priority (P1–P4 using the definitions below),
   route (Desktop, Apps, Identity, Network),
   and a one-line reason.
   
   Priority definitions:
   - P1: a whole site or a revenue-impacting service is down
   - P2: a team is blocked, or a single exec/VIP is fully blocked
   - P3: single user degraded, workaround exists
   - P4: request or question, no current impact
   
   Output a markdown table: number | category | priority | route | reason.
   If a ticket is ambiguous, set route to REVIEW instead of guessing.
   
   Tickets:
   [paste export]
   

   The REVIEW escape hatch matters: a model forced to always choose will confidently misroute the ambiguous 10%.

3. 3.

   Build the sampling-audit sheet once. Six columns: ticket, AI category, AI priority, AI route, correct? (Y/N), error type, action taken. Error type is one of: wrong category, over-prioritized, under-prioritized, wrong route, should-have-been-REVIEW. One sheet, one tab per week.

4. 4.

   Pull the daily sample: 10 random tickets plus every P1/P2. From a CSV export:

   Copy

   Import-Csv .\triage-export.csv |
     Get-Random -Count 10 |
     Select-Object number, short_description, ai_category, ai_priority, assignment_group
   

   Or straight from the ServiceNow table API:

   Copy

   curl -s -u "$SN_USER:$SN_PASS" \
     "https://yourinstance.service-now.com/api/now/table/incident?sysparm_query=sys_created_onONToday%40javascript%3Ags.beginningOfToday()%40javascript%3Ags.endOfToday()&sysparm_fields=number,short_description,category,priority,assignment_group&sysparm_limit=200" \
     | jq -r '.result[] | [.number,.priority,.category,.assignment_group.value] | @tsv' \
     | shuf -n 10
   

5. 5.

   Grade the sample in ten minutes. Open each sampled ticket, decide what the correct triage was, and fill the row. A good audit row looks like:

   Copy

   INC0048121 | AI: Software/P3/Apps | Correct: Access/P2/Identity
   error type: wrong route + under-prioritized
   action: rerouted to Identity, priority raised; pattern noted — "SSO" in
   subject keeps landing in Apps
   

6. 6.

   Act on patterns, not instances. Fix the sampled ticket immediately, but change the system only on repetition: more than 2 of 10 wrong on two consecutive days means retrain or tighten the prompt; one systemic class (every VPN ticket routed to Desktop) means a routing rule or a category definition fix; repeated under-prioritization of one user population means your P2 definition is too vague for the model — and probably for new hires too.

7. 7.

   Publish the weekly number. Misroute rate from the audit sheet goes in the team channel every Friday. A triage AI whose error rate nobody knows is a liability; one with a public, falling error rate is an asset you can defend to management.

Make it yours

• ServiceNow Predictive Intelligence shopschedule a monthly retrain of the classification solution definition after your audit sheet shows drift; feed corrected records back by fixing category on the ticket itself, since the next training run learns from it.
• Freshservice/Freddy shopuse the thumbs-up/down on Freddy suggestions religiously during the audit; it is the only feedback channel you have, and route fixes go into workflow automator rules, not the model.
• No-budget shoprun step 2 each morning on the overnight export, paste the table back as ticket updates via the bulk-edit UI or API, and keep the same audit sheet. The audit habit, not the vendor, is the transferable skill.

Quality check

• AI never finalizes P1/P2. High priorities are proposals; a human confirms within minutes, because both false alarms and missed majors are expensive.
• The sample is genuinely random. Cherry-picking easy tickets makes the audit theater. Use Get-Random/shuf, not your eyes.
• Test changes in sub-production first. Routing-rule and classifier changes go through your ServiceNow sub-prod instance or Freshservice sandbox before touching live intake.
• Keep the reason column honest. If you cannot say why the AI was wrong, mark it "unclear" — a pile of "unclear" rows is itself a signal the category scheme needs work.