Lab 10 · Capstone 1 — Streaming Chat Surface

Nadia's team is shipping their first AI feature: a help-panel chat. The prototype "streams" — text appears word by word, stakeholders applaud — but a look at the network tab shows one buffered response arriving whole, then animated client-side. Users wait the full generation time staring at a spinner; the typing effect is theater. This capstone builds the real thing, against a local mock model server (so it costs nothing and runs offline), and holds you to the only honest definition of streaming: bytes on the wire, progressively.

Time & tier

~120 minutes hands-on · Ship stage (maps to module M7, playbook PB-47) · $0 — everything runs locally against a mock LLM server you own. (Swapping the mock for a real provider later is a one-file change behind the same route.)

Setup

npm create vite@latest chat-lab -- --template react-ts
cd chat-lab && npm install
npm install express marked dompurify
npm install -D @types/express concurrently
mkdir -p server src/chat

The mock model + BFF server, server/index.mjs. It generates a canned-but-variable markdown answer token by token. It ships with the seeded transport — copy verbatim:

import express from "express";

const app = express();
app.use(express.json());

const ANSWER = [
  "Here's how to **paginate** the orders API:\n\n",
  "1. Request `GET /orders?cursor=<last-id>&limit=50`\n",
  "2. Read the `next_cursor` response header\n",
  "3. Stop when it returns `null`\n\n",
  "Example:\n\n```js\nlet cursor = null;\ndo {\n  const res = await fetch(`/orders?cursor=${cursor ?? ""}`);\n",
  "  const page = await res.json();\n  cursor = res.headers.get(\"next_cursor\");\n  render(page);\n} while (cursor);\n```\n\n",
  "Watch out for **cursor expiry** — cursors are valid for 10 minutes.",
];

async function* tokens() {
  for (const chunk of ANSWER)
    for (const tok of chunk.match(/.{1,4}/gs) ?? []) {
      await new Promise((r) => setTimeout(r, 30));
      yield tok;
    }
}

app.post("/api/chat", async (req, res) => {
  let full = "";
  for await (const tok of tokens()) full += tok;
  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  res.write(`data: ${JSON.stringify({ delta: full })}\n\n`);
  res.write("data: [DONE]\n\n");
  res.end();
});

app.listen(8787, () => console.log("mock LLM on :8787"));

Vite proxy in vite.config.ts so the client calls /api same-origin:

import { defineConfig } from "vite";
import react from "@vitejs/plugin-react";

export default defineConfig({
  plugins: [react()],
  server: { proxy: { "/api": "http://localhost:8787" } },
});

The starter chat panel, src/chat/ChatPanel.tsx — functional, and also seeded:

import { useEffect, useRef, useState } from "react";
import { marked } from "marked";
import DOMPurify from "dompurify";

type Msg = { role: "user" | "assistant"; content: string };

export function ChatPanel() {
  const [messages, setMessages] = useState<Msg[]>([]);
  const [input, setInput] = useState("");
  const [busy, setBusy] = useState(false);
  const logRef = useRef<HTMLDivElement>(null);

  useEffect(() => {
    if (logRef.current) logRef.current.scrollTop = logRef.current.scrollHeight;
  }, [messages]);

  async function send(prompt: string) {
    setMessages((m) => [...m, { role: "user", content: prompt }, { role: "assistant", content: "" }]);
    setBusy(true);
    const res = await fetch("/api/chat", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ prompt }),
    });
    const text = await res.text();
    const delta = JSON.parse(text.split("\n")[0].slice(6)).delta;
    setMessages((m) => {
      const copy = [...m];
      copy[copy.length - 1] = { role: "assistant", content: delta };
      return copy;
    });
    setBusy(false);
  }

  return (
    <div className="chat">
      <div ref={logRef} className="chat__log" role="log" aria-live="polite">
        {messages.map((msg, i) =>
          msg.role === "user" ? (
            <p key={i} className="chat__user">{msg.content}</p>
          ) : (
            <div key={i} className="chat__assistant"
              dangerouslySetInnerHTML={{ __html: DOMPurify.sanitize(marked.parse(msg.content) as string) }} />
          )
        )}
      </div>
      <form onSubmit={(e) => { e.preventDefault(); if (input.trim()) { send(input); setInput(""); } }}>
        <input aria-label="Message" value={input} onChange={(e) => setInput(e.target.value)} disabled={busy} />
        <button disabled={busy || !input.trim()}>Send</button>
      </form>
    </div>
  );
}

Run both: add to package.json scripts — "dev:all": "concurrently \"node server/index.mjs\" \"vite\"".

Guided steps

1. 1Run the starter and measure what you actually have.npm run dev:all, send a message, and open DevTools → Network → the /api/chat request. Look at the timing waterfall and the Response tab. Record in chat-log.md: when did the first byte arrive relative to total duration? Expected finding: one payload, arriving essentially all at once after multi-second silence — despite the SSE content type. Write down why (read the server route) before proceeding.
2. 2Fix the server: stream for real. Rewrite /api/chat to write an SSE event per token as it is produced — headers first (Content-Type: text/event-stream, Cache-Control: no-cache, and flush), then res.write(\data: ${JSON.stringify({ delta: tok })}\n\n`)inside thefor awaitloop, then[DONE]. Verify with curl -N -X POST localhost:8787/api/chat -H "Content-Type: application/json" -d '{}'` — you must see events trickling in live in the terminal. Do not touch the client until curl proves the wire is progressive.
3. 3Fix the client: consume the stream. Replace the await res.text() with a res.body.getReader() loop: decode chunks with TextDecoder, split on \n\n, parse each data: event, append delta to the last assistant message via functional setMessages. Handle the [DONE] sentinel. Expected: text now appears in the UI incrementally, and markdown re-parses progressively (partial markdown rendering imperfectly mid-stream is acceptable at this stage — note where it looks broken; Lab 11 and 12 build on this).
4. 4Add stop and regenerate. Wire an AbortController: Stop calls controller.abort(), the catch block preserves the partial content and appends a "stopped" marker; Regenerate re-sends the last user prompt, replacing the last assistant message. Both must be buttons with accessible names, disabled appropriately. Expected: stopping mid-stream leaves the partial answer visible, not a blank.
5. 5Diagnose the scroll behavior with a long transcript. Send four messages so the log scrolls. While a fifth streams, scroll up to re-read an earlier answer. Expected finding: the starter rips your scroll position back to the bottom on every token — the effect pins scrollTop unconditionally on each message update. Document the behavior, then fix it: track whether the user is near the bottom (e.g., scrollHeight - scrollTop - clientHeight < 40 captured on scroll), auto-follow only when they are, and show a "Jump to latest" affordance when they aren't.
6. 6Add error and retry. Kill the server mid-stream (Ctrl-C while a reply streams). Expected without handling: unhandled rejection, dangling empty message, dead input. Implement: catch stream errors, mark the message as failed with visible text, render a Retry button that re-sends, and re-enable the composer. Restart the server and prove Retry works.
7. 7Record the proof. DevTools → Network, send a message, and capture evidence that streaming is real: the EventStream tab of the /api/chat request showing events accumulating over time (screen-record it, or screenshot at two timestamps with visibly different event counts). Save as proof/ with the recording plus one sentence stating time-to-first-token vs. total time.
8. 8Write the capstone log. In chat-log.md: the two starter defects (mechanism, fix), TTFT before vs. after, and a checklist of the five required behaviors (stream, markdown+code, stop/regenerate, scroll anchoring, error retry) each marked with how you verified it.

Verify

• •
  Network evidence shows progressive events over time for a single /api/chat request — the recording in proof/ demonstrates it unambiguously.
• •
  curl -N against the server shows live token trickle (server-side streaming verified independently of the client).
• •
  Markdown renders with the fenced code block intact post-stream; sanitization (DOMPurify) still wraps every parse.
• •
  Stop preserves partial text; Regenerate replaces the last answer; both keyboard-operable.
• •
  Scrolled-up reading position survives an incoming stream; auto-follow resumes at bottom; "Jump to latest" appears when detached.
• •
  Killing the server mid-stream yields a visible failed state with working Retry — no console-only errors, no dead composer.

Self-grading rubric (pass = 7/10)

• Both starter seeds found by measurement (0–3): buffering discovered via network tab and scroll defect via the step-5 repro, before reading the key = 3.
• Transport correctness (0–3): server streams (curl-proven) and client consumes incrementally with clean [DONE]/abort/error paths = 3.
• UX completeness (0–2): stop/regenerate + scroll anchoring + retry all working = 2.
• Evidence quality (0–2): proof recording clearly shows progressive arrival with TTFT stated = 2.