Streaming Resilience: Batching, Cancellation, Resume

Tier: Advanced · Workflow area: AI features in the client

This playbook hardens the surface built in pb-47 against the four ways streams hurt in production: render thrash, zombie generations, lost streams and ambiguous partial failure.

When to use

• •
  A working chat/completion surface is showing its seams: typing jank on long replies, mobile CPU spikes, streams lost on refresh, or "it just stopped" bug reports.
• •
  You are building for scale or low-end devices, where per-token re-render cost is a real budget line.
• •
  Generations are long (reports, code) so mid-stream failure and abandonment are common, not edge, cases.

Skip resumability if generations are short and cheap to redo — batching and cancellation are still mandatory.

Prerequisites

• •
  The pb-47 baseline: useChat (or equivalent) against your own SSE endpoint, part-based rendering.
• •
  Ability to change the server route — resilience is a client-server contract, and half of this playbook lives in the route handler.
• •
  For resumable streams: a place to buffer stream state server-side (the AI SDK's resumable-stream path uses Redis or a compatible store).
• •
  A profiling habit: React DevTools Profiler or the browser performance panel open while you test on a throttled CPU.

The workflow

1. 1.

   Batch token application, don't render per chunk. Provider chunks can arrive at hundreds per second; applying each to state re-renders the transcript per token. Two levers, use both:

   • •
     Server/SDK smoothing: streamText supports chunk smoothing (experimental_transform: smoothStream()) to emit steadier, word-ish chunks.
   • •
     Client throttling: useChat accepts a throttle for UI updates (experimental_throttle: 50) so state applies at ~20fps regardless of chunk rate.

   Then stop the collateral re-renders: memoize each completed message component (React.memo keyed on message id + part count) so only the streaming message updates, and memoize markdown parsing per part — a naive implementation re-parses the entire transcript on every token.

2. 2.

   Make cancellation reach the provider, not just the UI. stop() aborting the client fetch is half the job; the server must propagate:

   Copy

   export async function POST(req: Request) {
     const result = streamText({
       model, messages,
       abortSignal: req.signal,          // provider call dies with the request
     });
     return result.toUIMessageStreamResponse();
   }
   

   Forward req.signal into the provider call so an abandoned tab or stop click stops token billing. Handle the abort on both sides: client marks the message as stopped-partial (kept, labeled), server's onAbort/onFinish persists the partial turn so refresh agrees with what the user saw.

3. 3.

   Decide the partial-failure contract explicitly. Streams have a nasty property: HTTP 200 is already sent when the provider errors mid-generation, so failure arrives inside the stream, not as a status code. Write the contract down and implement it:

   • •
     Error part in-band — the stream's error surfaces as a terminal error state on that message (the SDK emits an error part; render it), never a silent stop.
   • •
     Partial text is kept and labeled ("generation interrupted"), with retry regenerating the whole turn — do not attempt client-side "continue from where it stopped" unless the backend supports true continuation.
   • •
     The user's prompt is never lost — it is already persisted before generation starts.
   • •
     Retries are idempotent — regenerate replaces the failed assistant turn, keyed by message id, so double-clicks cannot fork the transcript.
4. 4.

   Add resumable streams for the flows that earn them. For long generations, wire the SDK's resumption path: the route buffers the stream under the chat id (Redis-backed resumable-stream), and the client, on mount, checks for an active stream for the conversation and re-attaches (resume: true in useChat). A refresh mid-generation then rejoins the live stream instead of orphaning it. Scope this to long-running surfaces — it adds infra (a store, stream GC) that short chats do not justify.

5. 5.

   Test the failure matrix, not the happy path. Simulate each: throttle CPU 6x and stream a 2,000-token reply (jank check); kill the network mid-stream (error-part rendering); refresh mid-generation (resume or clean partial); click stop and immediately regenerate (idempotency); background the tab for a minute mid-stream on mobile Safari (SSE suspension — verify reconnect or clean failure). Each scenario has a defined expected outcome from step 3's contract; anything undefined is a bug in the contract, not the code.

Tool variants

• AI SDK transport (default): everything above maps to shipped options — smoothing, throttle, abort forwarding, resumable streams — prefer configuration over custom code.
• Hand-rolled fetch + ReadableStream: you own chunk parsing, an rAF/interval batching buffer, AbortController wiring end-to-end, and reconnection; budget days, not hours, and steal the SDK's wire format.
• WebSocket transports: same contract, different failure modes (explicit close codes instead of dead SSE); the partial-failure matrix transfers unchanged.
• Mobile webviews / RN: streaming fetch support varies (classic RN needs a polyfill or native module for streaming bodies); test the matrix on the real runtime, not Chrome desktop.

Verification

• •
  Profiler shows only the streaming message re-rendering during generation; completed messages render zero times per token.
• •
  A 2,000-token stream on a 6x-throttled CPU keeps the input responsive and animation above ~30fps.
• •
  Stop click: provider-side generation terminates (verify via provider logs/billing or server logs), partial persists, refresh shows the same partial.
• •
  Mid-stream network kill produces the labeled error state with working retry; retry never duplicates a turn.
• •
  With resumption enabled: refresh mid-generation reattaches and the final transcript matches an uninterrupted run.

Pitfalls

• Render-thrash misdiagnosed as "model is slow". Users report slowness; the profiler shows 400 renders/sec. Measure before optimizing prompts.
• UI-only cancellation. Client aborts while the server happily streams to nobody — paying per token. Abort must propagate through req.signal to the provider call.
• Trusting HTTP status for stream health. The 200 lies; only in-band error parts tell the truth mid-stream.
• Auto-retry loops on stream errors. Blind client retries on a failing provider amplify outages and cost; retries are user-initiated or tightly budgeted.
• Resumability everywhere. The Redis buffer, GC and edge cases are real operational load; apply it only where generations are long enough to matter.
• Desktop-only testing. Mobile Safari's tab suspension and RN's streaming quirks are where stream bugs actually live.

Restricted-environment variant

All techniques here are provider-neutral and run against your own endpoint, so in-tenancy providers (Bedrock, Azure) change nothing client-side. One caveat: some corporate proxies and gateways buffer SSE responses, which silently defeats streaming — verify chunked delivery through the real network path early, and fall back to a "generate then render" non-streaming mode where the infrastructure cannot pass streams through.

Success criteria

• •
  Long generations stream smoothly on low-end hardware, with profiler-verified render isolation.
• •
  Zero zombie generations: every abandonment path (stop, tab close, refresh) terminates provider work.
• •
  Mid-stream failures produce labeled partials with working retry; "it just stopped" reports go to zero.
• •
  The partial-failure contract is a written document the backend team co-signed — and the failure matrix runs before each release touching the stream path.