Lab 10 · Virtual Service Agent, Deployed

Divya leads a six-person desk at a Noida SaaS company, and the CFO has asked why they don't "just get a bot." Divya's answer is this lab: not a slide about bots, a deployed one — configured in an afternoon on a free tier, then tested the way vendors never demo. Because the demo conversation is always the happy path, and the conversations that decide whether users trust the agent are the other kind: the question the KB cannot answer, the user who types like a human instead of a search query, and the frustrated person who needs the bot to get out of the way gracefully.

Two of the fifteen test conversations below are engineered to trap a badly configured agent in a loop. Your success condition for those two is not deflection — it is a warm handoff where the human picks up with full context and the user never repeats themselves.

Time & tier

~90 minutes hands-on · Core tier (maps to module M6). $0 — a free Jira Service Management tenant (virtual service agent included) or a Freshservice trial (Freddy). Either works; the lab is written platform-neutral. Plus any AI chat for drafting KB content.

Setup

Create these eight articles in your tenant's knowledge base — titles below, and use your AI to expand each into 6–10 lines of plausible article body (you are the review gate on what it writes, per lab-05). Note what is not covered: that gap is part of the test.

KB-01 Reset your password with the self-service portal
KB-02 Set up MFA on a new or replacement phone
KB-03 Connect to the VPN (Windows) and fix error 809
KB-04 Request access to a shared drive or mailbox
KB-05 Set up email on a mobile device (iOS/Android)
KB-06 Fix Teams sign-in and cache issues
KB-07 Request new software or a license via the portal
KB-08 Report a phishing or suspicious email

Save the pilot script as pilot-conversations.txt. Run each conversation against the live agent yourself, typing the user's lines verbatim:

C01 "How do I reset my password?"
C02 "I got a new phone and now I can't approve sign-ins"
C03 "vpn error 809 help"
C04 "My laptop screen has a crack, what do I do?"
C05 "I need access to the finance shared drive"
C06 "How do I get Photoshop installed?"
C07 "My email keeps asking me to sign in over and over"
    (user means the Outlook credential loop; KB-06 is closest)
C08 "I think I clicked a bad link in an email from 'DHL'"
C09 "Teams won't let me in, says something about credentials"
C10 "Set up email on my iPhone?"
C11 "The wifi in the Pune office is down for everyone on
    floor 3" (major incident, not a self-service topic)
C12 "How do I book a meeting room?" (not in KB)
C13 "This is the THIRD TIME I'm asking. Your bot keeps
    sending me the same password article. My account is NOT
    locked, my password is FINE, Outlook just keeps popping
    up a sign-in box. Get me a human."
C14 "hi"
C15 "I already tried everything in the VPN article. Error
    809 is still there. This is urgent, I have a client
    call in 20 minutes and honestly this bot has been
    useless so far."

Guided steps

1. 1Stand up the tenant and connect the KB. Sign up for the free JSM or Freshservice tenant, create the eight articles, and enable the virtual agent on the portal or a test channel. Expected output: the agent answers "how do I reset my password" with KB-01 content. Do not configure anything else yet — you want to see default behavior fail before you fix it.
2. 2Run the fifteen conversations cold and log everything. Type each user line, capture the agent's response, and score each conversation in pilot-log.md: resolved / escalated well / escalated badly / looped / hallucinated. Expected output at this stage: several failures — the out-of-KB questions and the two frustrated users (C13, C15) are where default configs loop. A clean first run usually means your test lines were too polite; check you typed them verbatim.
3. 3Configure intents deliberately. Create or tune intents mapping the KB topics, and pay attention to vocabulary: C07 and C09 describe credential problems in user language that must land on the right article, not keyword-match to the wrong one. Where the platform allows synonyms or training phrases, add the phrasings from the script. Re-run C07 and C09. Expected output: both reach relevant content without the user rephrasing.
4. 4Wire escalation with context handoff. Configure the escalation path: when the agent hands off, the created ticket must carry the conversation transcript, the articles already offered, and the user's own words — so the human never opens with "have you tried resetting your password?" Test by triggering an escalation and reading the resulting ticket as if you were the agent picking it up. Expected output: you could act on the ticket without asking the user to repeat anything.
5. 5Add frustration and repeat-contact triggers. Configure what your platform offers: sentiment or keyword triggers ("useless," "third time," all-caps), a repeat-of-same-intent counter, and an always-available "talk to a human" affordance. The rule you are implementing: two failed attempts or one frustration signal = offer a human, immediately, without another article.
6. 6Handle the out-of-scope classes. C04 (hardware damage), C11 (major incident), C12 (not in KB): each needs honest behavior — a clear "I can't help with this, here's what happens next" plus correct routing (C11 should raise priority, not offer the VPN article). Configure and re-test. Expected output: no hallucinated answers for topics the KB does not cover; C12 must not get a confidently invented meeting-room procedure.
7. 7Re-run the full fifteen. Log the after-scores next to the before-scores. The two frustrated conversations are pass/fail on three specifics: a human was offered within one exchange, the handoff ticket carried full context, and the bot did not send C13 the password article again.
8. 8Write the pilot report. One page in pilot-report.md: before/after scores, the conversations that improved and why, remaining failures you could not fix on the free tier (be honest — platform limits are findings, not embarrassments), and your go/no-go recommendation for a single-channel production pilot with before/after metrics you would track (contact volume, escalation rate, reopen rate — not just "deflection").

Verify

• •
  The agent is live on a free tenant with all eight KB articles connected and retrievable.
• •
  pilot-log.md scores all fifteen conversations before and after configuration.
• •
  C07 and C09 reach credential-relevant content despite non-technical user vocabulary.
• •
  C13 and C15 reach a human within one exchange of the frustration signal, with full-context tickets — verified by reading the ticket as the receiving agent.
• •
  C04, C11 and C12 produce honest, correctly routed non-answers; C11 is treated as higher-priority than a how-to.
• •
  pilot-report.md states before/after results, platform limits encountered, and a metrics-backed go/no-go.

Self-grading rubric (pass = 7/10): both frustration conversations pass all three handoff specifics (0–3) · vocabulary-mismatch conversations resolved (0–2) · out-of-scope trio honest and routed (0–2) · handoff ticket usable without user repetition (0–2) · pilot report metrics go beyond deflection (0–1).