An LLM feature without tracing is a black box that bills you. When quality drops you cannot say which prompt version regressed; when the invoice doubles you cannot say which endpoint did it.
Tier: Advanced · Workflow area: Building LLM Features
This playbook instruments every call — prompt version, tokens, latency, cost, and a quality signal — samples production traffic into your eval queue (pb-50), and then applies the two levers that actually cut bills: prompt caching and model routing. Instrument first, optimize second: every cost decision here is made against measured data.
docker compose up) or your existing OpenTelemetry stack; the schema below works in either.Define the trace schema — this is the contract everything else queries. One span per LLM call, nested under the request trace:
@dataclass
class LLMSpan:
trace_id: str # joins to the user request
feature: str # "triage" | "rag_answer" | "sql_gen"
model: str # exactly as returned by the response
prompt_version: str # e.g. "triage-v7" — version your system prompts
input_tokens: int
output_tokens: int
cache_read_tokens: int # cached-prefix reads (cheap tokens)
cache_write_tokens: int
latency_ms: int
ttft_ms: int | None # streaming features only
cost_usd: float # computed from pricing config
finish: str # end_turn | max_tokens | refusal | tool_use
error: str | None
quality: float | None # filled async, step 4
Emit it from one wrapper, not twenty call sites. Wrap the client once; every feature goes through it:
def traced_call(feature: str, prompt_version: str, **kwargs):
t0 = time.monotonic()
try:
resp = client.messages.create(**kwargs)
except anthropic.APIError as e:
emit_span(feature, prompt_version, error=type(e).__name__,
latency_ms=int((time.monotonic() - t0) * 1000)); raise
u = resp.usage
emit_span(feature, prompt_version, model=resp.model,
input_tokens=u.input_tokens, output_tokens=u.output_tokens,
cache_read_tokens=u.cache_read_input_tokens or 0,
cache_write_tokens=u.cache_creation_input_tokens or 0,
cost_usd=price(resp.model, u), finish=resp.stop_reason,
latency_ms=int((time.monotonic() - t0) * 1000))
return resp
Version every system prompt with a constant (TRIAGE_V7) that rides along in the span — "which prompt produced this" must be a query, never an archaeology project.
Build the three dashboards that answer real questions. (a) Cost: $/day by feature × model, tokens by feature, cache-hit ratio; (b) Latency: p50/p95 and TTFT by feature; (c) Health: error rate by type, finish=max_tokens rate (silent truncation), refusal rate. Alert on: daily cost > budget, cache-hit ratio dropping (a silent cache invalidator shipped — see below), and truncation rate > 1%.
Sample production into the eval queue. Nightly job: pull ~1% of spans plus all user-flagged outputs, run pb-50's judge over them, write quality back to the span. Now quality is a time series you can correlate with prompt_version and model — regressions become graphs instead of anecdotes, and the worst-scoring traces become new golden-dataset cases.
Cut costs with prompt caching — then verify it in the traces. Mark the stable prefix (system prompt, tool definitions, schema cards) with cache_control: {"type": "ephemeral"} on the last stable block. Cached reads bill at ~10% of input price. Caching is prefix-exact: a timestamp interpolated into the system prompt, unsorted JSON, or a per-user ID early in the prompt silently zeroes it. Your dashboard's cache-hit ratio is the detector — if cache_read_tokens stays at 0, diff two rendered prompts byte-by-byte.
Route by task difficulty — with eval evidence. Not every call needs the frontier model. Pick routing candidates from the data: features with high volume, low judge-score variance, and simple outputs (classification, short extraction) are candidates for a smaller model (e.g. claude-haiku-4-5 at a fraction of the cost). Process: run pb-50's suite for that feature on the candidate model; if the score holds within your threshold, route that feature and keep the big model for the hard features. Re-run the suite on both routes after every prompt change.
Close the loop monthly. Review: cost per feature trend, cache ratio, routing table, quality time series. Kill optimizations that degraded quality; promote ones that held.
Route with evidence, not intuition. The eval suite from pb-50 is what makes "use the cheaper model here" an engineering decision instead of a gamble — never downgrade a route without the suite's sign-off.
response.usage from the Responses API and check its cached-token fields — the schema above is provider-neutral by design.cost_usd from spans reconciles with the provider invoice within a few percent.prompt_version in the span, quality regressions cannot be attributed and rollbacks are guesswork. This one field is the cheapest high-value line in the schema.finish=max_tokens. A rising truncation rate is users silently getting cut-off answers — it is a quality incident wearing a cost metric's clothes.