LLM Observability: Tracing, Cost Dashboards, and Quality-Drift Alerts

Tier: Advanced · Workflow area: Serving & LLMOps

This playbook instruments an existing LLM app (your RAG service or agent from earlier playbooks) with OpenTelemetry/OpenInference tracing via self-hosted Arize Phoenix (or MLflow 3 tracing), builds the token-cost dashboard, and adds quality-drift alerting by running evals on sampled production traces.

When to use

• •
  Any LLM app past the demo stage — the first "it gave a weird answer yesterday" report you cannot investigate is the signal you waited too long.
• •
  You own cost or quality SLOs for an LLM feature and currently learn about breaches from users.
• •
  Prerequisite position: this is the observability capstone that makes pb-48's gateway data and your eval harness pay off jointly.

Prerequisites

• •
  A working LLM app you control (RAG service, agent) with reachable code.
• •
  pip install arize-phoenix openinference-instrumentation-<your-framework> — Phoenix self-hosts with no account; or MLflow ≥3 for its tracing UI if MLflow is already your shop standard.
• •
  Your eval harness (faithfulness/correctness scorers) from the evals module, runnable as a function over (input, context, output) triples.
• •
  Grafana + Prometheus optional, for the cost dashboard variant beyond Phoenix's built-ins.

The workflow

1. 1.

   Start Phoenix and instrument the app. phoenix serve gives a local UI and an OTLP collector endpoint. Auto-instrument your framework (OpenInference instrumentors exist for LangChain/LangGraph, LlamaIndex, OpenAI-compatible clients, Anthropic) so every request emits a trace with nested spans: retrieval, each LLM call with prompt/completion/token counts, tool executions, latency per span.

2. 2.

   Verify trace completeness on one request. Open a trace in the UI and check you can see: the user input, retrieved chunks with scores, the exact final prompt, the completion, token usage, and total latency. If a span is missing (custom code paths), add manual spans — an incomplete trace is a false sense of coverage.

3. 3.

   Build the cost view. Token counts per span × your price table (or gateway-reported cost from pb-48) → cost per request, aggregated by route/feature/day. Phoenix surfaces token stats; for stakeholder dashboards, export to Prometheus/Grafana or a scheduled notebook. The two charts that matter: daily spend by feature, and cost-per-request distribution (p50/p95 — the p95 tail is where prompt bloat and runaway agents show up).

4. 4.

   Add online eval sampling. A scheduled job (cron, every hour/day) pulls a sample of recent traces (e.g. 5–10%), runs your scorers on (input, retrieved context, output) — faithfulness and answer-relevance for RAG, task-specific rubrics otherwise — and logs scores back as trace annotations. Use a cheap-tier model for judge calls; sampling exists precisely so evaluation cost stays a rounding error.

5. 5.

   Alert on drift, not single failures. Compute a rolling baseline (e.g. 7-day mean faithfulness); alert when the daily sampled mean drops by more than a threshold (say 5 points) or p95 cost/latency breaches SLO. Route alerts to Slack. Single bad samples are noise; sustained shifts are retrieval rot, upstream model changes, or new traffic mixes.

6. 6.

   Close the loop with a triage drill. Inject a regression (swap in a worse retriever setting, or truncate context) and practice the diagnosis path: alert → dashboard → filtered traces → offending span → root cause. Time it; that duration is your real MTTR.

Tool variants

• MLflow 3 tracingreal-time trace logging plus registered scorers on traces; the right pick if MLflow already hosts your experiments and registry (pb-50).
• Langfuse (OSS)comparable self-hosted tracing with strong prompt-management integration.
• LangSmith / Arize (managed)same concepts, hosted; use when self-hosting is the constraint you want to delete.

Verification

• •
  One production-shaped request yields a complete trace: input → retrieval → prompt → completion → cost, no gaps.
• •
  Cost dashboard reconciles with provider/gateway billing within ~5% over a test window.
• •
  The injected regression fires the drift alert within one sampling cycle, and you traced it to the offending span in under 15 minutes.
• •
  Judge scores on sampled traces spot-checked against your own reading of 10 examples.

Pitfalls

• Tracing only the LLM call. Retrieval and tool spans are where RAG/agent failures live; instrument the whole chain or triage stays guesswork.
• Alerting on single-sample scores. LLM-judge noise guarantees false pages; alert on aggregates against a rolling baseline.
• Storing traces with PII indefinitely. Traces contain full prompts and retrieved documents — set retention and access controls like any sensitive log store.
• Judge model drift. Pin the judge model/version used for monitoring; silently upgrading the judge shifts your baseline and fakes a quality change.

Restricted-environment variant

Phoenix, Langfuse, MLflow, and Grafana all self-host — the full stack runs inside a client VPC with no data egress, which is exactly why the OSS path is taught here. Point judge calls at your self-hosted vLLM model. If even trace storage is sensitive, redact prompt/document payloads at the instrumentation layer and keep only structure + metrics; you lose drill-down depth but keep drift detection.

Success criteria

• •
  End-to-end tracing live on a real app with complete spans and token/cost capture.
• •
  Daily cost and quality views that reconcile with billing and with human spot-checks.
• •
  A drift alert that fired correctly in the injected-regression drill, with measured time-to-root-cause.
• •
  Sampling keeps monitoring cost under ~1–2% of serving cost — reported, not assumed.