Instrument LLM Traffic: TTFT, TPS, and Cost-per-Request with Langfuse

Tier: Advanced · Workflow area: Platform-for-AI: observability & cost

This playbook instruments the pb-45 gateway stack with self-hosted Langfuse, defines the metric vocabulary, load-tests to find your real TTFT/TPS envelope, and sets the alerts that catch cost and quality regressions.

When to use

• •
  LLM features are in production (or headed there) and "how much does this cost per request?" has no dashboard answer.
• •
  Users report "it's slow" and you can't distinguish queueing, prefill, or decode as the cause.
• •
  You need trace-level attribution: which team, feature, prompt version, and model produced each token of spend.

Skip standalone tooling only if your existing observability vendor's LLM module already gives you trace-level tokens/cost/TTFT — then apply this playbook's metrics and alerts there instead.

Prerequisites

• •
  The pb-45 LiteLLM gateway (or any point where all LLM traffic passes).
• •
  Self-hosted Langfuse (Docker Compose or Helm) with LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY created, and your instance URL (SDK env var LANGFUSE_BASE_URL; older SDK versions called it LANGFUSE_HOST).
• •
  The metric vocabulary, precisely: TTFT (request sent → first streamed token; queue + prefill), TPS/ITL (output tokens per second after the first; decode speed), E2E latency (TTFT + output_tokens ÷ TPS), cost-per-request (input_tokens × input_price + output_tokens × output_price).

The workflow

1. 1.

   Instrument at the gateway first. One config change covers every consumer — in litellm-config.yaml:

   Copy

   litellm_settings:
     success_callback: ["langfuse"]
     failure_callback: ["langfuse"]
   

   with the Langfuse keys and base URL in the proxy's environment. Every request now lands in Langfuse as a generation with model, tokens, computed cost, latency, and your key/team metadata. Verify field names against your pinned LiteLLM version.

2. 2.

   Add app-level traces where the workflow is multi-step. Gateway traces show calls; they can't show that three calls belong to one user action. In services with RAG/agent pipelines, use the Langfuse SDK (OpenTelemetry-based; @observe decorator or spans as context managers in Python) to wrap the workflow so retrieval, tool calls, and generations nest under one trace. Propagate user_id, session_id, and a prompt_version tag — these make regressions attributable.

3. 3.

   Load-test to find the envelope. Use an LLM-aware load generator (vLLM's vllm bench serve, genai-perf, or a k6 script that timestamps first-chunk arrival on streaming responses — plain HTTP timing misses TTFT). Sweep concurrency (1, 8, 16, 32...) with a realistic prompt/output length mix and record P50/P95 TTFT and per-request TPS at each step. The knee of the TTFT curve is your capacity number; publish it next to the SLO.

4. 4.

   Define SLOs on the right measurements. Interactive chat: P95 TTFT (e.g. < 1.5s) and minimum streaming TPS (reads-like-typing threshold, ~10+ tok/s). Batch: throughput and cost-per-request, not latency. Alert on burn rate like any other SLO.

5. 5.

   Set the regression alerts that errors won't catch.

   • •
     Cost: daily spend per team/feature vs trailing baseline (+X% alerts); cost-per-request P95 jump (a prompt change quietly doubling input tokens is the classic).
   • •
     Latency: P95 TTFT above SLO, sustained TPS drop (often a provider-side or fallback-model change).
   • •
     Quality proxies: error/refusal rate by taxonomy (4xx, 429, content-filter, timeout), fallback-served request rate (from pb-45), and — where you run them — Langfuse eval scores on sampled production traces (pairs with pb-52's eval discipline).
6. 6.

   Close the attribution loop. Build the two dashboards people actually use: (a) FinOps view — spend by team/feature/model, trend and forecast; (b) SRE view — TTFT/TPS/error taxonomies by model and route. Both derive from the same traces; check dashboard definitions into git.

Tool variants

• Langfuse (this playbook): OSS, self-hostable, OTel-based — the default client-safe path; also carries prompt management and evals you'll reuse in pb-52.
• Arize Phoenix: OSS alternative with strong OTel/OpenInference tracing; good in notebook-heavy ML orgs.
• Datadog LLM Observability / vendor APM modules: right when the org already lives in that vendor; insist on trace-level token/cost data, not just aggregate dashboards.
• Portkey analytics: built into the gateway if you chose Portkey in pb-45.

Verification

• •
  A single user action appears as one nested trace: retrieval span, tool spans, generation with tokens and cost.
• •
  Langfuse cost for a sampled day reconciles with the provider invoice / LiteLLM spend table within a few percent.
• •
  Load-test report exists with P95 TTFT and TPS at each concurrency step, and the SLO references it.
• •
  Each alert has fired at least once in a drill (inflate a test prompt, throttle a provider) and routed to the right owner.

Pitfalls

• Non-streaming benchmarks. Measuring only total latency hides TTFT entirely — users perceive TTFT; always test with streaming on.
• Prompt bodies in traces. Traces capture prompts/completions by default — apply masking/redaction config before pointing production client traffic at any observability store, and set retention deliberately.
• Averages. Token-count distributions are heavy-tailed; P50 cost-per-request looks fine while P99 requests burn the budget. Chart percentiles.
• Ignoring the fallback dimension. A latency improvement that coincides with fallback-to-smaller-model serving is a quality regression wearing a performance costume. Always segment metrics by served model.

Restricted-environment variant

• •
  Self-host Langfuse inside the same boundary as the gateway and models; for client-dedicated stacks, per-client instances keep trace data (which contains prompts = client data) inside each client's tenancy.
• •
  Enable prompt/completion masking for any trace store outside the client boundary; metadata-only tracing (tokens, latency, cost, no bodies) still delivers most of this playbook's value.
• •
  Load-test with synthetic prompts, never captured client traffic, unless the client's policy explicitly allows replay.

Success criteria

• •
  Any engineer can answer, from dashboards, in under a minute: what did feature X cost yesterday, and is TTFT within SLO right now?
• •
  A deliberately inflated prompt in staging triggered the cost-regression alert before a human noticed.
• •
  Capacity decisions (replicas, GPU class, provider mix) cite the load-test envelope, not vibes.
• •
  Post-incident reviews for LLM features attach traces, the same way service incidents attach APM links.