Building Support AI I: Assistants & RAG

Priya works L2 at a Bengaluru MSP, four years in, AZ-104 half-studied on her desk. In August her manager gives her the project she has been angling for: stand up a virtual service agent for the MSP's biggest client on their Freshservice tenant. The platform makes the start almost embarrassingly easy — connect the KB, enable the agent, pick channels. Atlassian advertises build-to-live in under 30 minutes for its JSM equivalent, and that is roughly what Priya's first version takes.

The pilot numbers after two weeks look spectacular: 71% of conversations "handled without escalation." Then the client's IT lead forwards her a transcript. An employee locked out during payroll week asked for help four different ways; the bot answered each rephrase with the same confident, slightly-wrong KB article about password expiry (the real issue was account lockout after a phone change — an article that existed but was never retrieved); the employee asked for a human twice; the bot offered the article a fifth time. The employee gave up and called the desk — which is why the conversation counted as "handled without escalation." The bot's best number was measuring its worst behavior.

Priya's fix took three weeks and is this module: escalation paths with real handoff, retrieval tuned against a golden set of actual user questions, honest "I don't know" behavior, and an eval habit that measures what users experienced rather than what the dashboard implies. The second pilot's headline number was lower — 54% — and every point of it was real. This module builds both halves: the platform skills that get a bot live, and the engineering-plus-honesty that makes it worth having live.

Deploying a Virtual Service Agent End-to-End

The platform path — JSM virtual service agent, Freddy AI Agent, Copilot Studio — is genuinely low-code, and the free tiers (JSM free plan, Freshservice trial) mean you can do everything in this section this week. The work divides into five stages, and the config is the easy two of them:

1. 1Connect the knowledge. Point the agent at the KB — which is why Module 3 precedes this one. An agent over a rotted KB automates wrong answers; run the health audit first, always.
2. 2Define the intents that matter. Not fifty intents — the top contact types from your actual ticket data: password/lockout, access requests, VPN, the client's three chronic apps. Each intent gets its resolution flow: KB answer, guided steps, or a fulfillment action.
3. 3Design the escalation paths. Before launch, not after the transcript embarrasses you. Covered in its own section below, because it is the difference between deflection and abandonment.
4. 4Pilot on one channel. One team or one channel (the IT Slack/Teams channel is ideal), 2–4 weeks, with before/after measurement of contact volume, resolution, satisfaction — and transcript review, which is where Priya's payroll-week failure would have surfaced in week one.
5. 5Measure honestly, then expand. Module 7 owns the measurement discipline; the preview: "handled without escalation" is not resolution, and a bot that loses users to give-up inherits their anger at the desk anyway.

The Pipeline Under Everything: RAG

Every service-desk AI — Freddy, Now Assist's Q&A, your future custom bot — is retrieval-augmented generation underneath: chunk the KB, embed the chunks, retrieve by similarity at question time, generate an answer grounded in the retrieved chunks, with citations, and refuse when nothing sufficient was found. The Common Core gave you RAG literacy; this module makes it a support-desk skill, and Lab 11 makes you build the whole pipeline yourself — Ollama for the local model, an open-source stack (LangChain/LlamaIndex-class), a free vector store (Chroma/pgvector-class), your sample KB in, a citation-grounded chatbot out, no budget and no data leaving your machine.

Building it yourself, once, is the point — not because you will replace Freddy, but because every configuration decision in every platform bot is a RAG decision wearing a settings page:

• Chunking is answer qualitychunks that split mid-procedure produce answers that end mid-procedure. Module 3's chunk-friendly authoring is the fix applied at the source; chunk size and overlap are the fix applied in the pipeline.
• Retrieval is the usual failure pointwhen a bot answers wrongly despite a correct article existing, the article usually wasn't retrieved (vocabulary mismatch, bad chunking), or lost the similarity race to a stale near-duplicate.
• Grounding discipline is what makes it support-gradethe generation step must be constrained to retrieved content, cite its sources, and refuse when retrieval came back thin. A bot that answers from the model's general knowledge is a liability wearing your desk's name.
• Local-first matters hereclient KBs and ticket corpora are client data; the Ollama path runs the entire pipeline on your machine, which is why it is this track's flagship build and the DPDP-clean default for sensitive corpora.

Retrieval Quality Engineering: The Golden Set

"The bot can't find the article that exists" is the #1 deployment complaint, and it is an engineering problem with an engineering method. You need one artifact and one loop.

The artifact is the golden set: 30–50 real user questions — mined from actual tickets, in actual user vocabulary, including the hard cases (vocabulary mismatches, questions whose answer spans two articles, questions the KB cannot answer where refusal is the correct behavior) — each paired with the article/chunk that should ground the answer. Lab 12 provides one; for your production desk you will build your own, and it becomes the desk's most valuable AI asset: the same set that tunes your bot also evaluates every vendor claim in Module 7.

The loop is eval-driven fixing, with failures classified before they are fixed:

The classification discipline is what separates engineering from thrashing: teams that skip it re-chunk everything when the actual problem was three missing articles, or rewrite prompts when retrieval never surfaced the right text. Measure, classify, fix the class, re-measure. Lab 12's artifact — before/after scores with a failure-taxonomy sheet showing which fix moved which failures — is exactly the evidence an employer reads as "this person can run a support-AI program."

Escalation Design That Respects Users

Now the payroll-week transcript, done right. The industry evidence is blunt: bots that never escalate post great deflection and terrible satisfaction — users abandon, rage-click, and arrive at the human desk angrier than if the bot had never existed. Escalation is not the bot failing; it is the bot working as designed. Design it with four mechanisms:

• Confidence thresholds with honest refusalbelow threshold, the bot says "I don't have a reliable answer for this" and routes — it does not present its best guess with full confidence. An honest "I don't know" preserves trust; a confident wrong answer spends it.
• Frustration detectionrepeated rephrasing, negative sentiment, an explicit "let me talk to a person": each is an immediate escalation trigger. The second request for a human is not an intent to deflect harder; it is the user telling you the interaction has already failed.
• Warm handoff with full contextescalation transfers the conversation, not just the user: what was asked, what was tried, what the bot retrieved. Forcing users to repeat everything to the human is the single most-hated support-bot behavior and it is entirely a design choice.
• Loop capsa hard limit on same-answer repetitions. Priya's bot offered the same article five times because nothing counted the repetitions. Two strikes, then escalate.

Action-Enabled Agents: When the Bot Does Things

The next step past Q&A is agents that act: create the ticket, reset the password via API, check license availability, trigger the Module 4 remediation script. Freddy's AI Agent Studio, Rovo agents and custom builds (Graph API, MCP-connected tools) all offer it, and it is where deflection gets real — a password reset actually executed is a resolution, not an answer. It is also where the desk inherits agent-safety engineering, and the Common Core's rules arrive with sharper teeth because the acting credential is an admin credential:

1. 1Scoped credentials per action. The password-reset action holds password-reset permission — not the service account that can do everything. One credential per action class, least privilege by construction.
2. 2Per-action risk classing. Read-only checks (license availability): agent may act. Reversible user-scoped actions (password reset with identity verification): act with logging and rate limits. Anything touching groups, permissions, data: approval-gated or absent. This is Module 4's execution-boundary table, now applied to a bot the whole company talks to.
3. 3Audit logging as a feature, not an afterthought. Every action: who asked, what verification, what executed, what result. When something goes wrong — and identity-verification edge cases guarantee eventually something will — the log is the difference between an incident and a mystery.
4. 4Identity verification proportional to the action. A bot that resets passwords on request, in chat, is a social-engineering front door. The verification flow for each action is a security design decision; make it with the security team, not alone.

Module 7's governance matrix formalizes all of this into the signable document; build nothing action-enabled before that matrix exists in at least draft form.

Rollout Is Change Management, Not Configuration

The quiet statistic behind failed assistant projects: adoption failure kills more deployments than model failure. A technically sound bot that nobody uses — or that users try once during a bad week-one and never return to — fails exactly as completely as one that answers wrongly. Treat multi-channel rollout as a change-management project: channel-appropriate behavior (Teams/Slack answers are conversational and short; portal answers can be article-length), a launch that sets honest expectations ("it will hand you to a human when it can't help" beats "meet your new AI assistant that can do anything"), visible early wins on the top-3 contact types, usage analytics watched weekly, and a feedback path users can see acting on their complaints. Priya's second pilot spent as much effort on the client's launch comms as on retrieval tuning; the retention curve was the reward.

Try It Now

1. 1Mine your intents. From your last 200 tickets (or your desk's category report), rank the top 10 contact types by frequency. Mark each: bot-answerable from the KB, bot-actionable via API, or human-only. This is your pilot scope, evidence-based.
2. 2Build a mini golden set. Write 10 real user questions from real tickets — in the users' words — and pair each with the KB article that should answer it. Run them through any bot or RAG tool you have access to; score answers and classify failures as retrieval, generation, or content.
3. 3Red-team an escalation path. On any chatbot you can reach (your desk's, or a vendor demo), play the frustrated user: ask, rephrase three times, request a human twice. Log at which turn — if any — you reached one, and what context survived the handoff.