Lab 13 · AI Cost Architecture Retrofit

Sameer's month started with a CFO screenshot: the GCC's internal helpdesk copilot, piloted to 400 employees, ran 47× over its cost forecast (synthetic numbers, real pattern). The post-mortem found no bug — just an architecture with no cost layers: every request, including the 40th identical "how do I reset my password," went to the frontier model with full context. Cost, like security, is a property you design in; retrofitting is exactly this lab. The order of layers matters, the thresholds matter more, and the only honest way to claim savings is to replay a workload and measure hit rates per layer — which is what you will do, at ₹0, with local models standing in for priced ones.

Time & tier

~90 minutes hands-on · Advanced tier (maps to module M7). ₹0 path: all calls go to local Ollama models and costs are computed arithmetically from a rate card you fill in from today's provider pricing pages. No real API spend required.

Setup

mkdir lab-13 && cd lab-13
python -m venv .venv && source .venv/bin/activate
pip install ollama numpy
ollama pull llama3.1:8b        # plays "frontier"
ollama pull llama3.2:1b        # plays "small/cheap"
ollama pull nomic-embed-text   # embeddings for the semantic cache

Create rate_card.md and fill it in now, from the provider pricing pages, today — the numbers below are deliberately blank because stated LLM prices go stale and this track never asserts them from memory:

# Rate card (filled from provider pricing pages on: ____-__-__)
| Line              | Unit          | Rate (₹) | Source URL |
|-------------------|---------------|----------|------------|
| frontier input    | per 1M tokens |          |            |
| frontier output   | per 1M tokens |          |            |
| small input       | per 1M tokens |          |            |
| small output      | per 1M tokens |          |            |
| embeddings        | per 1M tokens |          |            |
Pick any one real provider pair (frontier + small tier) and record the date.

Generate the workload — repetition and near-duplicates are built in, because real helpdesk traffic has both. Create gen_workload.py:

# gen_workload.py — 200 requests: exact repeats + near-duplicates + unique
import json, random
random.seed(42)
BASE = [
    "How do I reset my password?",
    "How do I reset my MFA device?",            # near the one above - trap pair
    "What is the guest wifi password policy?",
    "How do I request a new laptop?",
    "How do I claim travel reimbursement?",
    "Where do I download the VPN client?",
    "How do I book a conference room in Tower B?",
    "What are the office holidays this quarter?",
    "How do I raise a payroll discrepancy ticket?",
    "How do I get access to the analytics warehouse?",
    "What is the notice period policy for contractors?",
    "How do I renew my building access card?",
]
PARA = {
    "How do I reset my password?":
        ["I forgot my password, what now?", "Password reset steps please"],
    "Where do I download the VPN client?":
        ["VPN client download link?", "Need the VPN installer, where is it?"],
    "How do I claim travel reimbursement?":
        ["Steps to file a travel expense claim?", "Travel claim process?"],
}
reqs = []
for i in range(200):
    q = random.choice(BASE)
    if q in PARA and random.random() < 0.5:
        q = random.choice(PARA[q])
    reqs.append({"id": i, "question": q})
json.dump(reqs, open("workload.json", "w"), indent=1)
print("wrote 200 requests,", len({r['question'] for r in reqs}), "distinct strings")

Run it. And here is the naive app — read it and estimate, before running anything, what fraction of its 200 frontier calls are avoidable. Create naive_app.py:

# naive_app.py — every request goes to the frontier model; no layers
import json, time, ollama
FRONTIER = "llama3.1:8b"
SYSTEM = ("You are the internal helpdesk copilot. Answer briefly and "
          "practically for an enterprise employee.")

def call(model, q):
    t0 = time.time()
    r = ollama.chat(model=model, messages=[
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": q}])
    return (r["message"]["content"],
            r.get("prompt_eval_count", 0), r.get("eval_count", 0),
            time.time() - t0)

if __name__ == "__main__":
    reqs = json.load(open("workload.json"))
    tot_in = tot_out = tot_t = 0.0
    for req in reqs:
        _, tin, tout, dt = call(FRONTIER, req["question"])
        tot_in += tin; tot_out += tout; tot_t += dt
        print(f"[{req['id']:3d}] frontier {dt:4.1f}s")
    print(json.dumps({"frontier_calls": len(reqs), "in_tokens": tot_in,
                      "out_tokens": tot_out, "avg_latency_s": tot_t/len(reqs)}))

Guided steps

Step 1 — Baseline run and baseline bill (15 min)

Run python naive_app.py | tee runs/baseline.log (expect several minutes — that pain is data). Record: frontier calls (200), total input/output tokens, average latency. Convert tokens to rupees with your rate card and write the number down as the baseline monthly bill under a stated scaling assumption (e.g., "this workload × 150 business days" — synthetic, but explicit). Also compute the theoretical floor from gen_workload.py's output: with only ~17 distinct question strings and ~19 distinct intents, what is the minimum number of frontier calls a perfect cache stack would make?

✅ Checkpoint: runs/baseline.log exists; you have a baseline cost figure and a written estimate of the avoidable fraction.

Step 2 — Layer 1: exact-match cache (15 min)

Create layered_app.py by copying the naive app and adding the first layer — a dict keyed on the normalized question (q.lower().strip() with whitespace collapsed and trailing punctuation dropped). On hit: return the stored answer, count exact_hit, spend zero tokens. Re-run the replay. Exact repeats are common in this workload by construction, so expect a meaningful hit rate — but note what the layer cannot catch: "Password reset steps please" misses a cache warmed by "How do I reset my password?".

✅ Checkpoint: replay shows a nonzero exact_hits count, frontier calls dropped accordingly, and paraphrases still miss.

Step 3 — Layer 2: semantic cache with a threshold you must choose (20 min)

Add the embedding layer behind the exact cache:

# add to layered_app.py — semantic cache (checked only after exact miss)
import numpy as np, ollama
SEM = []            # list of (embedding, question, answer)
THRESHOLD = 0.92    # you will tune this - it is the whole lesson

def embed(q):
    v = np.array(ollama.embed(model="nomic-embed-text",
                              input=q)["embeddings"][0])
    return v / np.linalg.norm(v)

def semantic_lookup(q):
    if not SEM: return None
    v = embed(q)
    sims = [float(v @ e) for e, _, _ in SEM]
    best = int(np.argmax(sims))
    if sims[best] >= THRESHOLD:
        return SEM[best][1], SEM[best][2], sims[best]
    return None

Run the replay twice: once at THRESHOLD = 0.92, once at 0.80. For each run, log every semantic hit as (incoming question, matched question, similarity). Then audit the 0.80 run's hit list by hand: find the hits where the matched question asks something different — the workload's trap pair ("reset my password" vs "reset my MFA device") is embedded close enough to collide at loose thresholds on most embedding models. Count these as false_hits and compute the false-hit rate alongside the hit rate.

✅ Checkpoint: both runs logged; you can report hit rate and false-hit rate per threshold, and the trap pair appears in your 0.80 audit (or you have verified your embedding model keeps them apart and said so).

Step 4 — Layer 3: the two-tier router (20 min)

Below both caches, route what remains. Default cheap, escalate on a rule:

# add to layered_app.py — two-tier router under the caches
SMALL, FRONTIER = "llama3.2:1b", "llama3.1:8b"
ESCALATE_MARKERS = ("policy", "notice period", "payroll", "compliance")

def route(q):
    # Rule: short factual how-tos -> small model; policy/HR-sensitive or long,
    # multi-part questions -> frontier. State your rule; measure it later.
    ql = q.lower()
    if any(m in ql for m in ESCALATE_MARKERS) or len(q.split()) > 24:
        return FRONTIER, "frontier"
    return SMALL, "small"

The rule above is a starting point, not gospel — a confidence-gated router (small model answers, self-assessed low confidence escalates) is the stronger production pattern; note it in your report as the next iteration. Wire the full chain — exact → semantic (0.92) → router — and make every layer increment its counter: exact_hits, sem_hits, small_calls, frontier_calls, plus tokens per model tier.

✅ Checkpoint: full-chain replay runs green; the four counters sum to 200; frontier calls are now a small minority.

Step 5 — The savings report (20 min)

Write savings_report.md comparing baseline vs full chain, with a per-layer table:

| Layer          | Hits | Hit rate | Tokens avoided | ₹ avoided (rate card) |
|----------------|------|----------|----------------|-----------------------|
| exact cache    |      |          |                |                       |
| semantic cache |      |          |                |                       |
| small tier     |      |  (of routed) |            | (delta vs frontier)   |
| frontier       |      |  residual    |     -      |          -            |

Include: latency before/after (caches answer in milliseconds — cost layers are also latency layers), the false-hit analysis with your chosen threshold and why, cache-invalidation policy (what TTL for HR answers? what happens when the VPN link changes?), and the honest caveats — a 12-intent synthetic workload flatters caches relative to open-ended traffic, so state the hit rates as workload-dependent, not universal. Close with the CFO paragraph: baseline bill, retrofit bill, and which layer earned what.

✅ Checkpoint: savings_report.md has the per-layer table with all four hit rates, priced deltas from your rate card, and at least two stated caveats.

Verification

• Counters reconcileexact + semantic + small + frontier = 200 on the full-chain replay.
• Threshold experimenthit rate and false-hit rate reported at two thresholds, with the trap pair explicitly dispositioned.
• Priced honestlyevery rupee figure traces to the rate card, and the rate card carries today's date and source URLs.
• Latency told the truthreport shows per-layer latency, not just cost.

Completion artifact

The lab-13/ repo: gen_workload.py, naive_app.py, layered_app.py, rate_card.md, runs/ logs, and savings_report.md. Self-grading rubric (pass = 8/10):

• Per-layer instrumentation (0–3): four counters, reconciling to 200, with per-layer token accounting = 3.
• Threshold judgment (0–3): both thresholds run, false hits audited by hand, final threshold chosen with a stated reason = 3.
• Pricing discipline (0–2): rate card filled from sources today; no price asserted from memory anywhere in the report = 2.
• Report candor (0–2): caveats about synthetic workload and cache invalidation present; CFO paragraph readable by a CFO = 2.

Client-safe variant

The ₹0 path is already client-safe: workload, models, and embeddings are all local, and the only external touch is reading public pricing pages. On a real engagement, the sensitive artifact is the workload — real user questions are user data, so replay from the client's logs only inside their environment, redact identifiers before any prompt leaves a controlled tenancy, and present the savings report with hit rates and percentages rather than raw question text. The rate-card discipline transfers unchanged: architects who quote model prices from memory in a client cost model are shipping the category's signature failure mode.

Stuck?

• •
  Baseline run is painfully slow: that is the lab working — but you may cut the workload to 100 requests (edit the range) as long as before/after use the same file.
• •
  prompt_eval_count missing from responses: some Ollama versions omit counts on cached model state — restart Ollama, or approximate with len(text)//4 and say so in the report.
• •
  Semantic cache never hits: confirm you normalized embeddings (unit vectors) and are comparing with dot product; unnormalized vectors make 0.92 unreachable.
• •
  Semantic cache hits everything: your embedding vectors are being compared to themselves — check you only insert after a miss and answer, not before lookup.
• •
  The trap pair doesn't collide at 0.80: embedding models differ — record the actual similarity you measured for the pair and pick a threshold with margin below it; the lesson is the audit, not the specific number.

Answer key (don't read before attempting)

The seeded traps and expected shape of results:

1. 1The trap pair is the point of Step 3. "How do I reset my password?" vs "How do I reset my MFA device?" — high lexical and semantic overlap, materially different answers, and serving the password answer to the MFA question is a wrong answer delivered confidently at cache speed. At 0.80 most embedding models collide them; at 0.92 they separate. The general rule: semantic-cache thresholds are set by auditing false hits on your own traffic, and a cache without a false-hit metric is unmeasured risk, not savings.
2. 2Expected magnitudes (workload-dependent, not universal): with 12 base intents, ~17 distinct strings, and 200 draws — exact cache typically absorbs a large majority of traffic (most draws repeat a prior string), the semantic cache catches most paraphrases (the three PARA intents), and the router splits the ~17 first-time questions with only the policy/HR-marked ones reaching frontier. Single-digit frontier calls out of 200 is the right ballpark; if you saw ~90% "savings," check the false-hit audit before celebrating.
3. 3Layer order is load-bearing. Router-before-cache means paying a model call to answer something the cache already knew; semantic-before-exact wastes an embedding call on identical strings. Cheapest-check-first is the same principle as Lab 01's gate ordering in the DevOps track — and the reason the report needs per-layer hit rates is that a mis-ordered stack still shows a healthy blended number.
4. 4What the retrofit still lacks (name these to score full report candor): cache invalidation/TTL (the VPN answer changes when IT changes the link), per-user/authz scoping (cached answers must not leak across permission boundaries), provider-side prompt caching as a fourth layer for shared system-prompt prefixes, and a confidence-gated router replacing the keyword rule. Design-time cost architecture means these were decided before launch — which is the sentence Sameer's post-mortem ends with.