Streaming LLM Responses Through Your Backend: SSE Relay, Cancellation, Failure Contract

Tier: Core · Workflow area: Building LLM Features

Streaming fixes perceived latency, but it moves complexity into your backend: you become a relay between the provider's stream and your client's, and you now own cancellation, mid-stream failures, and the question of what the frontend should do when a stream dies at token 400. This playbook builds the relay with an explicit contract for all three.

When to use

• •
  Any user-facing generation longer than ~2 seconds: chat, drafting, summarization panels.
• •
  Long outputs where a non-streaming request risks hitting HTTP timeouts.
• •
  You need usage/cost accounting per response and incremental delivery — streaming gives you both.

Skip streaming for machine-to-machine calls returning structured JSON (pb-44) — partial JSON is useless to a consumer, and buffering defeats the point.

Prerequisites

• •
  pb-44's foundations (timeouts, error taxonomy, redaction) — streaming layers on top.
• •
  pip install fastapi anthropic sse-starlette (or your stack's SSE primitive).
• •
  A frontend contract agreed with whoever renders it: event names for delta, done, and error.

The workflow

1. 1.

   Define the wire contract first. Three SSE event types, versioned like any API:

   • •
     delta — {"text": "..."} incremental tokens
   • •
     done — {"usage": {"input_tokens": n, "output_tokens": n}, "finish": "end_turn"}
   • •
     error — {"code": "upstream_error", "recoverable": true} — the frontend keeps partial text, shows a retry affordance.

   Writing this down is the deliverable the frontend team actually needs from you.

2. 2.

   Build the relay with a passthrough generator. Consume the provider stream, re-emit your contract:

   Copy

   import json, anthropic
   from fastapi import FastAPI, Request
   from sse_starlette.sse import EventSourceResponse
   
   app = FastAPI()
   client = anthropic.Anthropic(timeout=60.0)
   
   @app.post("/chat/stream")
   async def chat_stream(request: Request):
       body = await request.json()
   
       def relay():
           try:
               with client.messages.stream(
                   model="claude-opus-4-8",
                   max_tokens=64000,
                   messages=[{"role": "user", "content": body["message"]}],
               ) as stream:
                   for text in stream.text_stream:
                       yield {"event": "delta", "data": json.dumps({"text": text})}
                   final = stream.get_final_message()
                   yield {"event": "done", "data": json.dumps({
                       "usage": {"input_tokens": final.usage.input_tokens,
                                 "output_tokens": final.usage.output_tokens},
                       "finish": final.stop_reason,
                   })}
           except anthropic.APIError:
               yield {"event": "error",
                      "data": json.dumps({"code": "upstream_error", "recoverable": True})}
   
       return EventSourceResponse(relay())
   

3. 3.

   Handle client disconnects — stop paying for abandoned streams. When the user closes the tab, stop consuming the upstream: check await request.is_disconnected() in an async variant (or rely on generator teardown closing the with block, which aborts the provider stream). Log abandoned streams separately — a high abandon rate is a product signal, not just a cost leak.

4. 4.

   Buffer deliberately, in one place. Per-token SSE events can overwhelm mobile clients and proxies. Coalesce to ~50–100ms flushes in the relay if needed — never in the frontend, and never so aggressively that time-to-first-token exceeds a second. First token fast is the entire point.

5. 5.

   Account at stream end. get_final_message() gives you the complete message and usage even though you streamed — log model, prompt version, tokens, latency-to-first-token, total latency, and finish reason. finish: "max_tokens" at any meaningful rate means your cap is truncating real answers.

6. 6.

   Test the ugly paths. Kill the upstream mid-stream (drop network) and verify the client receives error with partial text intact; verify a closed client tab actually cancels the upstream call (watch the provider dashboard / logs); verify proxies in your path (nginx, ALB) are not buffering SSE — X-Accel-Buffering: no, no gzip on the stream route.

Tool variants

• Anthropic (shown):client.messages.stream(...) context manager; text_stream for deltas, get_final_message() for accounting.
• OpenAI: Responses API with stream=True yields typed events (response.output_text.delta, response.completed); same relay pattern, map their events to your contract.
• WebSocket instead of SSE: justified only for bidirectional needs (mid-generation user actions, voice). SSE survives proxies/CDNs better and reconnects natively via EventSource — default to SSE.
• Node: identical architecture with ReadableStream/res.write on Express or route handlers on Next.js; the contract from step 1 is unchanged.

Verification

• •
  Time-to-first-token p95 under ~1.5s from your endpoint (measure at the relay, not the provider).
• •
  Mid-stream upstream kill → client gets error, keeps partial text, retry works.
• •
  Tab-close test → provider-side stream is cancelled (confirm via usage logs: output tokens stop growing).
• •
  curl -N against the endpoint through your full proxy chain shows tokens arriving incrementally, not one buffered blob.
• •
  Usage totals from done events reconcile with the provider dashboard within a few percent over a day.

Pitfalls

• Proxy buffering. The #1 field failure: works locally, arrives as one blob in prod because nginx/ALB/CDN buffered the response. Disable buffering and compression on the stream route explicitly.
• No failure contract. If the frontend team learns about mid-stream errors in production, that is a backend design failure. The contract in step 1 comes first.
• Leaking provider event shapes to the client. Relay your contract, not the vendor's raw events — otherwise a provider SDK upgrade becomes a frontend breaking change.
• Ignoring cancellation. Abandoned generations silently become 10–20% of your bill on chat-shaped features.
• Streaming structured JSON to code. If the consumer parses the result, use pb-44's non-streaming shape; stream only human-watched text.

Restricted-environment variant

• Client-tenancy deploymentsthe relay pattern is unchanged on Bedrock (AnthropicBedrockMantle) or Azure OpenAI streaming — both support SSE-style streams inside the tenant boundary.
• Local/offlineOllama streams over its OpenAI-compatible API; the relay code and contract are identical, which makes local an excellent integration-test double for CI.
• Redaction and output safetyredaction (pb-44 step 2) still applies to the inbound user message; the streamed output should be covered by your output-safety checks (pb-52) before broad rollout.

Success criteria

• •
  A documented three-event contract the frontend implemented against without reading your code.
• •
  First token in ~1s, smooth delivery through the real proxy chain, graceful partial-failure UX.
• •
  Abandoned streams cancel upstream; usage logs reconcile with the provider bill.
• •
  One incident-drill run: you can tell "provider died", "client left", and "completed" apart from logs alone.