AI features get deactivated after launch more often than any other feature class — Applause's 2026 survey has 44.1% of orgs pulling one back — and the killers are rarely accuracy. They are cost overruns, intolerable latency, and providers failing in ways nobody rehearsed. Budget all three, and gate on the budgets.
Tier: Core · Workflow area: Testing AI systems — judges, agents & robustness
This playbook establishes token-spend and latency budgets with Langfuse cost tracking, load-tests streaming endpoints with k6, and runs fault-injection drills — provider outage, brownout, and the silent model swap — so degradation behavior is a tested feature, not a production surprise.
Any LLM feature heading to production with per-call costs and a latency-sensitive UX. Before capacity commitments or pricing decisions. After incidents where the provider degraded and the feature failed noisily — or worse, silently. When agents with variable step counts make per-task cost unpredictable.
Internal prototypes with capped usage and tolerant users — measure, but gating is premature. Batch offline pipelines where latency is irrelevant — keep the cost budget, drop the latency ceremony.
Define the three budgets in writing. Per feature, one page:
Feature: support-answer-drafting v2.4 budgets
Cost: p50 <= $0.011 / task, p95 <= $0.04 / task
(agent step cap 6; context cap 12k tokens)
Latency: TTFT p95 <= 1.2s, full response p95 <= 8s
Resilience: provider outage -> degraded template mode within 5s,
zero user-facing 500s; recovery automatic
Percentiles, not averages — LLM latency and agent cost are long-tailed, and the p95 is what users and finance actually experience. TTFT (time to first token) is the latency number that matters for streaming UX; total duration matters for downstream timeouts.
Instrument cost per task, not per call. One user task may be five model calls across an agent loop. Group calls under a trace, let Langfuse roll up token costs, and report the task-level distribution:
Langfuse: traces for support-answer-drafting, last 7d (n=4,812)
cost/task p50 $0.009 p95 $0.031 max $0.87
tokens/task p50 6.1k p95 19.4k
FLAG: 14 traces > $0.25 — all show retrieval loop re-fetching
the same document up to 9x
The max column is where agent bugs live; the $0.87 trace is a looping regression that a mean would have hidden completely.
Load-test the streaming path with k6. Measure TTFT explicitly — default HTTP metrics report total duration and miss the number your UX depends on:
// k6 excerpt: TTFT via a custom Trend metric
const ttft = new Trend('ttft', true);
const res = http.post(url, payload, { timeout: '60s' });
ttft.add(res.timings.waiting); // first byte for streaming endpoint
check(res, { 'status 200': (r) => r.status === 200 });
k6 run --vus 20 --duration 5m llm_stream_test.js
Run at realistic concurrency and watch for the rate-limit cliff: provider 429s typically begin at some concurrency step, and your p95 doubles in one step rather than degrading smoothly. Find the cliff in staging, then set production concurrency limits below it.
Drill provider outage. Point the provider base URL through Toxiproxy (or a mock gateway) and cut it mid-load:
toxiproxy-cli create llm_upstream -l :8474 -u api.provider.com:443
toxiproxy-cli toxic add llm_upstream -t timeout -a timeout=0
Assert the product behavior, not the HTTP behavior: users get the degraded experience you designed (cached answer, template response, honest "assistant unavailable"), within the budgeted seconds, with no raw stack traces; recovery is automatic when the toxic is removed; on-call got exactly one alert, not four hundred.
Drill the brownout, not just the blackout. Total outage is the easy case. Add latency (+3s per call) and a 30% error rate — the half-alive provider is where retry storms are born. Assert: retries are capped with backoff and jitter, timeouts stack sanely across an agent's steps (six sequential 30s timeouts is a 3-minute user wait), and cost during the brownout stays bounded — retries are paid tokens, and a retry storm is a cost incident and an availability incident simultaneously.
Test the silent model swap. Providers update and alias models; your app config can drift. Two checks: a canary assertion in CI comparing the responding model identifier against the pinned expectation, and a weekly golden-run comparing quality-eval scores and cost/latency profile against the stored baseline — an unannounced backend change shows up as a profile shift even when the model name has not changed.
Wire budgets into CI as regression gates. Every prompt or model PR runs the golden set with cost and latency recorded; the gate fails on budget regression, with the same override discipline as a failing functional test:
Budget gate: FAIL
cost/task p95: $0.052 vs budget $0.04 (+30%)
cause: prompt v2.5 adds 2 few-shot examples (+3.1k tokens/call)
options: trim examples, or PM signs revised budget in this PR
That last line is the discipline: budgets change by decision, recorded in the PR — never by drift.
Accuracy decides whether an AI feature is good; cost, latency and failure behavior decide whether it survives. The 44.1% that got switched off mostly passed their quality evals — they never rehearsed the meter, the spinner, or the outage.
Put the cost-per-task trend on the same dashboard as the pass-rate trend, in front of the same audience. The features that die post-launch die in the gap between those two charts — closing the gap is this playbook in one sentence.