Lab 15 · Observability Capstone: Traces, Cost, and One Real Regression

This is the on-call skill: when quality drops at 2 a.m., the traces are all you have.

Time & tier

Advanced · ~2.5 hours hands-on · API spend under $2.

Setup

Python 3.11+, an Anthropic API key, and:

mkdir lab-15 && cd lab-15 && python -m venv .venv && source .venv/bin/activate
pip install arize-phoenix openinference-instrumentation-anthropic anthropic pydantic
export ANTHROPIC_API_KEY="sk-ant-..."

Reused assets (with fallbacks for skipped labs): corpus.py (the Orbitly policy docs + retriever — copy the self-contained snippet from lab 12's Setup), golden_v1.jsonl (questions — generate with lab 09's make_golden.py), and judge.py (lab 09's calibrated judge). All three are fully inline in those labs; five minutes of copy-paste if you skipped them.

The app under observation is a compact RAG support bot. Create rag_app.py:

import os, json, anthropic
from phoenix.otel import register
from corpus import retrieve

tracer_provider = register(project_name=f"support-bot-{os.getenv('APP_VERSION', 'a')}",
                           auto_instrument=True)      # picks up the Anthropic instrumentor
tracer = tracer_provider.get_tracer(__name__)

try:
    from deploy_b import transform_docs               # installed by apply_regression.py
except ImportError:
    transform_docs = lambda docs: docs

client = anthropic.Anthropic(max_retries=3)
MODEL = os.getenv("MODEL", "claude-haiku-4-5")        # cheap tier for replay volume — deliberate

def answer(question: str) -> dict:
    with tracer.start_as_current_span("rag.answer") as span:
        docs = retrieve(question)
        if os.getenv("APP_VERSION", "a") == "b":
            docs = transform_docs(docs)
        span.set_attribute("retrieval.documents", json.dumps(docs))
        ctx = "\n".join(docs) or "(none)"
        resp = client.messages.create(
            model=MODEL, max_tokens=300,
            system=("Answer from the policy context only. If it doesn't cover the question, "
                    "say so and offer to escalate. Quote numbers exactly."),
            messages=[{"role": "user", "content": f"<context>{ctx}</context>\n\n{question}"}],
        )
        text = next(b.text for b in resp.content if b.type == "text")
        return {"question": question, "context": ctx, "answer": text,
                "in_tok": resp.usage.input_tokens, "out_tok": resp.usage.output_tokens}

Guided steps

1. 1.

   Launch Phoenix and take baseline traffic. In a second terminal: phoenix serve — UI at http://localhost:6006. Then replay production-like traffic against version A. Create replay.py:

   Copy

   import json, random, time, sys
   from rag_app import answer
   
   random.seed(23)
   items = [json.loads(l) for l in open("golden_v1.jsonl")]
   questions = [it["question"] for it in random.sample(items, 20)] * 2   # 40 calls
   
   version = sys.argv[1] if len(sys.argv) > 1 else "a"
   with open(f"results_{version}.jsonl", "w") as f:
       for q in questions:
           t0 = time.perf_counter()
           r = answer(q); r["latency_s"] = round(time.perf_counter() - t0, 3)
           f.write(json.dumps(r) + "\n")
   print(f"replayed {len(questions)} calls -> results_{version}.jsonl")
   

   Run python replay.py a. In Phoenix, open project support-bot-a: each trace shows your rag.answer span (with the retrieval payload attribute) wrapping the auto-instrumented Anthropic span (prompt, completion, token counts, latency). Click through five traces until the anatomy is boring — that familiarity is what you'll lean on later.

2. 2.

   Evals on sampled traces. Nobody judges 100% of production traffic; sample and score. Create sample_eval.py:

   Copy

   import json, random, sys
   from judge import judge
   
   version = sys.argv[1]
   rows = [json.loads(l) for l in open(f"results_{version}.jsonl")]
   random.seed(5)
   sample = random.sample(rows, 12)
   verdicts = []
   for r in sample:
       v = judge(r["question"], r["context"], r["answer"])
       verdicts.append({"question": r["question"], "verdict": v.verdict, "quote": v.quote})
       print(v.verdict, "|", r["question"][:60])
   rate = sum(v["verdict"] == "pass" for v in verdicts) / len(verdicts)
   print(f"sampled faithfulness ({version}): {rate:.0%}")
   json.dump(verdicts, open(f"evals_{version}.json", "w"), indent=2)
   

   Run it for version A and record the rate — this is your quality baseline, produced by the judge whose kappa you actually know (lab 09). Optionally push the verdicts into Phoenix as span annotations so quality lives next to latency in one UI.

3. 3.

   The cost dashboard. Create cost.py — per-version tokens, spend, and latency percentiles from the replay logs (cross-check the token numbers against what Phoenix shows on the traces):

   Copy

   import json, sys, statistics
   PRICE_IN, PRICE_OUT = 1.00, 5.00       # $/MTok for the bot model tier
   
   for version in sys.argv[1:]:
       rows = [json.loads(l) for l in open(f"results_{version}.jsonl")]
       tin = sum(r["in_tok"] for r in rows); tout = sum(r["out_tok"] for r in rows)
       lat = sorted(r["latency_s"] for r in rows)
       cost = (tin * PRICE_IN + tout * PRICE_OUT) / 1e6
       print(f"v{version}: calls={len(rows)} in_tok={tin} out_tok={tout} "
             f"cost=${cost:.4f} (${cost/len(rows)*1000:.2f}/1k answers) "
             f"lat p50={statistics.median(lat):.2f}s p95={lat[int(0.95*len(lat))]:.2f}s")
   

4. 4.

   Deploy the regression — without looking at it. A "bad deploy" ships to version B. Create apply_regression.py, run it once, and honor-system: do not open deploy_b.py — on-call you, staring at traces, wouldn't have the diff either.

   Copy

   import base64
   BLOB = "ZGVmIHRyYW5zZm9ybV9kb2NzKGRvY3MpOgogICAgcmV0dXJuIFtkLnNwbGl0KCIuICIpWzBdICsgIi4iIGZvciBkIGluIGRvY3NdCg=="
   open("deploy_b.py", "w").write(base64.b64decode(BLOB).decode())
   print("version B deployed")
   

   Then replay production on the new version: APP_VERSION=b python replay.py b, and score it: python sample_eval.py b, python cost.py a b.

5. 5.

   Triage from traces alone. The rules: Phoenix UI, evals_*.json, and cost.py output only — no source diffing. Work it like an incident:

   • •
     Detect: the sampled faithfulness rate for B dropped hard (expect roughly 90%+ → ~50–70%). That alone pages you.
   • •
     Scope: in the judged sample, which questions fail? Cluster them — you should find failures concentrated where the answer depends on a fact that lives mid-policy (grace periods, expiry windows, credit formulas), while first-fact questions still pass.
   • •
     Correlate: cost.py shows B's input tokens are dramatically lower than A's at identical questions and unchanged output style. Cheaper and worse is a context regression signature, not a model regression.
   • •
     Localize: open failing B traces in Phoenix and read the retrieval.documents attribute next to the same question's A trace. The comparison is the smoking gun — state precisely what changed about the retrieved context.
   • •
     Explain the misses: with the supporting fact gone, the grounded prompt forces the model toward either "I don't have that information" or an invented specific; your judge quotes tell you which happened per item.
6. 6.

   Write the RCA memo. RCA.md, one page: Symptom (metric + magnitude + when), Blast radius (which question types, with two trace IDs each for A/B), Root cause (what the traces prove changed, stated mechanically), Fix (roll back version B), Verification (re-run replay + sampled evals on the fix; numbers restored), Prevention (this exact diff would have been caught by lab 10's CI gate — say which check and threshold). Then actually do the verification: delete deploy_b.py, replay as fix, and show the faithfulness rate recover.

Verification

• •
  Phoenix shows separate trace sets for versions A and B with retrieval payloads and token counts on every trace.
• •
  Sampled-eval faithfulness recorded for A, B, and the fix; B is measurably worse and the fix recovers to baseline.
• •
  cost.py output shows the input-token drop for B and a cost table per 1k answers.
• •
  RCA.md names the true root cause with paired trace IDs as evidence — written before reading the answer key.
• •
  You did not open deploy_b.py before the memo (honor system, but it is your own training you'd be cheating).

Completion artifact

RCA.md + screenshots (or exported trace links) of one A/B trace pair and the Phoenix project view + evals_{a,b,fix}.json + the cost.py table. Self-grade (pass at 4+):

• Full tracing: tracing captures retrieval payloads, not just LLM I/O.
• Measured quality: quality measured by the calibrated judge on a defined sample, not anecdote.
• Cost evidence: cost/token evidence used in triage.
• Proven root cause: root cause stated mechanically and proven with paired traces.
• Prevention wired in: prevention section ties back to a concrete CI gate.

Stuck?

• No traces in Phoenix:register() must run before the first Anthropic call (import order in rag_app.py matters), and phoenix serve must be up first; check the collector endpoint printed by register().
• Traces land in one merged project: the project name is read from APP_VERSION at import time — run each replay in a fresh process with the env var set, as shown.
• Judge and Phoenix token numbers disagree: Phoenix reads the API's usage block, cost.py reads the same values you logged — if they differ you edited the log fields; trust the span.
• Faithfulness barely drops on B: your sampled questions skew toward first-sentence facts — raise the sample to 20 or seed differently, and note the sampling lesson in the memo.

Answer key

deploy_b.py decodes to a one-liner that truncates every retrieved policy to its first sentence: return [d.split(". ")[0] + "." for d in docs] — a stand-in for the classic bad chunking/truncation deploy. The trace signature you should have found: (1) input tokens per call drop sharply on B with identical questions; (2) the retrieval.documents attribute shows single-sentence contexts where A had full policies; (3) judged failures cluster on questions whose supporting fact lives after the first sentence (export-link expiry, 2FA grace period, SLA credit math, refund eligibility caveats), producing either unsupported invented specifics or unnecessary "I don't have that information" refusals; (4) latency and output length stay roughly flat, ruling out model or capacity causes. Full credit is naming "retrieved context is being truncated to its first sentence" — not merely "retrieval got worse."

Client-safe variant

Self-hosted Phoenix is the client-safe default: traces contain prompts, retrieved client documents, and customer questions, so the collector must live inside the client's network (their VM or cluster, phoenix serve behind their auth) with no SaaS observability export unless legal signs off. Add a redaction hook before spans leave the app (mask emails/IDs in the retrieval payload attribute — the PII patterns from your production stack, not this lab's toy ones), set a trace-retention window in writing, and scope UI access like production-log access. The triage methodology, memo format, and sampled-eval design are yours to reuse across clients; the traces never are.