Every service-desk AI product is the same machine under the paint: chunk the knowledge base, embed it, retrieve what matches the question, and make the model answer only from what was retrieved — with citations, and a refusal when nothing relevant exists. Build it once yourself and no vendor demo will ever mystify you again.
Tier: Core · Workflow area: Assistants & RAG · ~45 minutes
This playbook builds a working retrieval-augmented chatbot over your IT knowledge base on a local, no-data-leaves-the-building stack — Ollama for models, a LangChain/LlamaIndex-class framework, a Chroma/pgvector-class vector store — with the two behaviors that separate trustworthy from dangerous: verifiable citations, and refuse-when-not-found. (UC-40)
Stand up the local stack. Install Ollama and pull a chat model and an embedding model (ollama pull llama3.1 and ollama pull nomic-embed-text — or current equivalents). Create a Python environment with your framework and vector store (for example pip install langchain langchain-community langchain-ollama chromadb — a LlamaIndex or raw-pgvector build follows identical steps). Export 20–50 KB articles as markdown or text files into a kb/ folder — enough to be real, small enough to inspect.
Chunk with the metadata that makes citations possible. Split articles into overlapping chunks (start near 800 characters with 100 overlap; pb-39 tunes this), and attach source (filename/article ID) and title metadata to every chunk — a citation is only as good as the metadata you kept at ingest:
from langchain_community.document_loaders import DirectoryLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
docs = DirectoryLoader("kb/", glob="**/*.md").load()
chunks = RecursiveCharacterTextSplitter(
chunk_size=800, chunk_overlap=100).split_documents(docs)
Embed and index. Every chunk becomes a vector; similar meaning lands near similar meaning, which is the entire trick:
from langchain_ollama import OllamaEmbeddings
from langchain_chroma import Chroma
store = Chroma.from_documents(
chunks, OllamaEmbeddings(model="nomic-embed-text"),
persist_directory="./kb-index")
Prove retrieval before adding generation. Query the store raw and read what comes back:
for d in store.similarity_search("how do I set up VPN on a Mac?", k=4):
print(d.metadata["source"], "::", d.page_content[:80])
If the top chunks are not the VPN article, no prompt downstream can save you — retrieval quality is the ceiling of the whole system. Fix chunking or content before proceeding.
Add generation with the grounding contract in the system prompt. This prompt is the product:
You are an IT support assistant. Answer ONLY from the context
chunks provided below. Rules:
- After each factual claim, cite the source article in brackets,
e.g. [vpn-mac-setup.md]
- If the context does not contain the answer, reply exactly:
"I don't have this in the knowledge base — please raise a ticket."
Do not answer from general knowledge. Do not guess.
- If chunks conflict, say so and cite both.
Context: {retrieved chunks with their source metadata}
Question: {user question}
Wire it as: embed question → retrieve top-k → format chunks with sources into the prompt → generate.
Test the refusal behavior as hard as the answers. Ask ten questions the KB covers and ten it cannot ("how do I configure the espresso machine?", plausible-but-absent IT questions like a product you don't run). The covered set should answer with citations that check out when you open the file; the absent set should refuse every time. A model that answers confidently from general knowledge on the absent set has a broken grounding contract — tighten the prompt (and lower the temperature) until refusal is reliable. This test is the difference between a grounded assistant and an eloquent liability.
Demo it with the citations open. Show the team: question, answer, and the cited article side by side — then one refusal. The refusal demo lands harder than the answers; it is the behavior nobody's chatbot experience has taught them to expect.
The local stack's privacy guarantee holds only while it stays local — the moment you swap in a cloud model API or host the index elsewhere, your KB's contents are governed by that provider's data terms, so re-check before any such swap, and keep credentials and secrets out of the KB you index.
Chunk, embed, retrieve, ground — with citations that resolve and a refusal when retrieval comes back empty; that contract, not the model, is what makes a service-desk AI trustworthy, and now you can inspect any vendor's version of it from the inside.