First Production LLM Endpoint: Structured Outputs, Timeouts, Retries, Fallbacks

Tier: Core · Workflow area: Building LLM Features

This playbook wraps a model API behind your own endpoint with all five, in runnable shape.

When to use

• •
  You are shipping your product's first LLM-backed feature (classification, extraction, summarization) behind an existing backend.
• •
  A prototype notebook call needs to become a service other teams depend on.
• •
  You inherited an LLM integration that parses model output with regex and prays.

Not for agentic multi-step features (see pb-46) or streaming UX (see pb-45) — this is the request/response foundation both build on.

Prerequisites

• •
  Python 3.11+, FastAPI (or your stack's equivalent — the shape ports directly), and an API key: pip install fastapi anthropic pydantic.
• •
  Budget guard: a few dollars of credits is enough to build and load-test this.
• •
  Agreement on the output contract with whoever consumes the endpoint — the schema is the API.

The workflow

1. 1.

   Define the contract as a schema, not a prompt instruction. Pydantic model = single source of truth for the model output and your response type:

   Copy

   from pydantic import BaseModel, Field
   from typing import Literal
   
   class TicketTriage(BaseModel):
       category: Literal["billing", "bug", "how_to", "account", "other"]
       severity: Literal["p0", "p1", "p2", "p3"]
       summary: str = Field(description="One sentence, no PII")
       needs_human: bool
   

2. 2.

   Redact before you send. PII hygiene happens on your side of the wire — deterministically, before the SDK call:

   Copy

   import re
   
   PATTERNS = [
       (re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+"), "<EMAIL>"),
       (re.compile(r"\+?\d[\d\s()-]{8,}\d"), "<PHONE>"),
       (re.compile(r"\b(?:\d[ -]*?){13,19}\b"), "<CARD>"),
   ]
   
   def redact(text: str) -> str:
       for pattern, token in PATTERNS:
           text = pattern.sub(token, text)
       return text
   

3. 3.

   Make the call with schema enforcement, tight timeout, and bounded retries. The Anthropic SDK retries 429/5xx with backoff itself — configure it, don't reimplement it:

   Copy

   import anthropic
   
   client = anthropic.Anthropic(timeout=20.0, max_retries=2)
   
   SYSTEM = "You triage support tickets for a B2B SaaS. Be conservative: \
   if unsure of severity, pick the higher one and set needs_human=true."
   
   def triage(ticket_text: str) -> TicketTriage:
       response = client.messages.parse(
           model="claude-opus-4-8",
           max_tokens=1024,
           system=SYSTEM,
           messages=[{"role": "user", "content": redact(ticket_text)}],
           output_format=TicketTriage,
       )
       return response.parsed_output  # validated TicketTriage instance
   

4. 4.

   Map failures to an explicit error taxonomy — with a fallback model. Decide per error class: retry, fall back, or fail:

   Copy

   from fastapi import FastAPI, HTTPException
   
   app = FastAPI()
   FALLBACK_MODEL = "claude-sonnet-4-6"
   
   @app.post("/triage", response_model=TicketTriage)
   def triage_endpoint(body: dict):
       try:
           return triage(body["text"])
       except anthropic.RateLimitError:
           raise HTTPException(429, "Upstream busy, retry later")  # surface, don't spin
       except anthropic.APIStatusError as e:
           if e.status_code >= 500:            # provider incident: try the fallback once
               return triage_with_model(body["text"], FALLBACK_MODEL)
           raise HTTPException(502, "LLM upstream error")
       except anthropic.APIConnectionError:
           raise HTTPException(503, "LLM upstream unreachable")
   

5. 5.

   Log what you will need at 2am. Per request: model, prompt version (tag your SYSTEM string with a version constant), input/output tokens (response.usage), latency, and whether fallback fired. This is the seed of pb-51's observability.

6. 6.

   Load-test the failure paths, not just the happy path. Point the client at a mock that returns 429s and 500s; verify your endpoint degrades the way step 4 promises. Then a real-traffic soak: p95 latency, token cost per request.

Tool variants

• Anthropic (shown):client.messages.parse(...) with a Pydantic output_format returns parsed_output pre-validated.
• OpenAI: same shape on the Responses API — client.responses.parse(model="gpt-5.6", input=[...], text_format=TicketTriage), read response.output_parsed. Chat Completions uses response_format={"type": "json_schema", ...}.
• Bedrock / Azure OpenAI: identical code through the provider clients (AnthropicBedrockMantle(aws_region=...), Azure's OpenAI client) — the common route in enterprise/client tenancies. Check feature parity: not every structured-output mode exists on every platform.
• Node/TypeScript:@anthropic-ai/sdk + Zod (zodOutputFormat), or openai + zod — same contract-first pattern.

Verification

• Feed 20 real (redacted) tickets100% parse into the schema; spot-check that category/severity judgments are defensible.
• Kill your network mid-requestendpoint returns your 503, not a stack trace, within the timeout.
• Force a 429 (tiny rate limit or mock)SDK retries are visible in logs, then your 429 surfaces cleanly.
• Redaction under real inputsgrep an hour of request logs for @ and 10+ digit runs; redaction holds.

Pitfalls

• Parsing prose instead of enforcing schema. "Respond in JSON" in the prompt is a suggestion; output_format/text_format with a strict schema is a guarantee. Never json.loads free text in production.
• Retrying 4xx. A 400 will fail identically forever; retrying it burns budget and hides the bug. Only 429/5xx/network are retryable — the SDKs already know this.
• Unbounded max_tokens with no timeout. A pathological input can hold your worker for minutes. Timeout at the client, cap tokens at the request.
• Redacting after logging. The redaction in step 2 is useless if raw input already landed in your request logs — audit the whole path.
• Silent fallback quality drift. The fallback model answers differently. Tag responses with the serving model so quality regressions are attributable.

Restricted-environment variant

• Client policy forbids external LLM APIsrun the identical code against Bedrock or Azure OpenAI inside the client's cloud tenancy — the SDK swap is a constructor change, the playbook is unchanged.
• Nothing may leave the network at allOllama serving a local model exposes an OpenAI-compatible endpoint; keep the schema validation on your side (local models fail schema more often — validate and retry once, then route to a human queue).
• DPDP/GDPR contextstreat step 2 as mandatory scope, log redacted text only, and document the data flow for your DPO before launch.

Success criteria

• •
  An endpoint whose responses are schema-valid 100% of the time or fail with a taxonomy error — never a malformed body.
• •
  Provider outage simulation: graceful 429/503s and a working fallback, zero stack traces to callers.
• •
  Per-request cost and latency known from logs (p50/p95), and a redaction step you can show your security review.
• •
  A teammate can consume the endpoint from the OpenAPI schema alone, without reading a single prompt.