Bulletproof Structured Outputs: Schema-Enforced LLM Extraction

Tier: Core · Workflow area: Structured Outputs & LLM APIs

The output is an extraction function you can put behind a production endpoint and trust.

When to use

• •
  Any pipeline where LLM output feeds code, a database, or a downstream model — not a human reader.
• •
  You are replacing regex/rule-based parsers with LLM extraction and need parity-or-better reliability.
• •
  An existing prompt-only JSON extractor is throwing JSONDecodeError or silently dropping fields in production.

Skip this for exploratory notebook work where you will eyeball every output anyway.

Prerequisites

• •
  Python 3.10+, pydantic>=2, anthropic (or openai), optionally instructor and outlines.
• •
  An API key with a small budget — the full playbook runs in well under $2 of API spend on a default-tier model.
• •
  20–50 real examples of your messy input (support tickets, logs, free-text fields) and a written definition of the fields you need.

The workflow

1. 1.

   Define the schema as a Pydantic model, not a prompt paragraph. Field types, enums, and descriptions are your contract:

   Copy

   from pydantic import BaseModel, Field
   from typing import Literal
   
   class Ticket(BaseModel):
       product: str = Field(description="Product name as mentioned, verbatim")
       severity: Literal["low", "medium", "high", "critical"]
       churn_risk: bool
       summary: str = Field(description="One sentence, max 20 words")
   

   Structured-output APIs require additionalProperties: false and do not support numeric/length constraints server-side — the SDKs strip those and validate them client-side, so keep Field(ge=..., max_length=...) for the client-side pass.

2. 2.

   Layer 1 — schema-enforced generation. Use the API's native parse method so the schema is enforced at decode time, not hoped for:

   Copy

   import anthropic
   client = anthropic.Anthropic()
   
   resp = client.messages.parse(
       model="claude-opus-4-8",
       max_tokens=1024,
       messages=[{"role": "user", "content": f"Extract ticket fields:\n\n{raw_text}"}],
       output_format=Ticket,
   )
   ticket = resp.parsed_output  # a validated Ticket instance
   

3. 3.

   Layer 2 — validation-retry with Instructor. For semantic constraints the schema cannot express (e.g. "summary must not copy the input verbatim"), add a Pydantic validator and let Instructor feed the validation error back to the model:

   Copy

   import instructor, anthropic
   client = instructor.from_anthropic(anthropic.Anthropic())
   
   ticket = client.messages.create(
       model="claude-opus-4-8",
       max_tokens=1024,
       messages=[{"role": "user", "content": raw_text}],
       response_model=Ticket,
       max_retries=2,   # re-prompts with the validator's error message
   )
   

4. 4.

   Layer 3 — constrained decoding fallback (local/open-weights). When the input cannot leave your infrastructure, Outlines masks invalid tokens at generation time so the output is grammatically guaranteed to match the schema:

   Copy

   import outlines
   model = outlines.models.transformers("Qwen/Qwen2.5-7B-Instruct")
   generator = outlines.generate.json(model, Ticket)
   ticket = generator(prompt)
   

5. 5.

   Measure before shipping. Run all three layers over your 20–50 sample inputs and log: parse-success rate, per-field accuracy against a hand-labeled subset, retries per call, and latency. Structured outputs guarantee shape, not truth — field-level accuracy is the number that matters.

Tool variants

• Anthropic API:client.messages.parse(..., output_format=Model) → .parsed_output; raw form is output_config={"format": {"type": "json_schema", "schema": ...}}. For tool-calling pipelines, strict: true on the tool definition guarantees tool_use.input validates against the schema.
• OpenAI API:client.responses.parse(model=..., input=[...], text_format=Model) → response.output_parsed (Responses API), or client.chat.completions.parse(..., response_format=Model) → .choices[0].message.parsed.
• Instructor: one wrapper (from_anthropic / from_openai / from_provider) gives the same response_model + max_retries interface across providers — useful when you must stay vendor-neutral.
• Outlines / vLLM guided decoding: for self-hosted models; vLLM accepts guided_json=<schema> per request.

Verification

• •
  100% parse-success across your sample set at Layer 1 (anything less means the schema itself is invalid for the API — check for unsupported constraints).
• •
  Field-level accuracy ≥ your rule-based baseline on the hand-labeled subset; report per-field, not aggregate.
• •
  Retry count logged per call — a rising retry rate is your early-warning signal for input drift.
• •
  One deliberately malformed/adversarial input (empty string, 50-page paste, wrong language) returns a controlled failure, not an exception that kills the batch.

Pitfalls

• Enums with an escape hatch. A Literal["a","b","c"] forces the model to pick one even when none applies. Add "unknown" to every enum and treat its frequency as a data-quality metric.
• Optional fields silently absorbing failures.str | None everywhere gives you 100% parse success and empty rows. Make fields required unless absence is genuinely meaningful.
• Validating shape and calling it accuracy. Teams celebrate 0% parse errors while 15% of severity labels are wrong. Always pair schema enforcement with a labeled accuracy sample.
• Schema churn invalidating caches. Providers cache compiled schemas (first request is slower); constantly regenerating schemas dynamically forfeits both the cache and reproducibility. Version your schemas like code.
• Retrying without a cap. Instructor's max_retries multiplies cost and latency; cap at 2 and route persistent failures to a dead-letter queue for review.

Restricted-environment variant

For client data that cannot reach a hosted API (standard for India services-company engagements):

• •
  Run Layer 3 as the primary path: Outlines or vLLM guided decoding with a 7–8B instruct model. Qwen2.5-7B-class models handle typed extraction well; fits a free Kaggle/Colab T4 (16 GB) with 4-bit quantization.
• •
  Keep the identical Pydantic schema and eval harness — only the generation call changes, so your accuracy numbers stay comparable across hosted and local paths.
• •
  If a hosted API is permitted after redaction, strip names, IDs, and financial figures with placeholders before the call and re-join locally.

Success criteria

• •
  An extraction function with three explicit reliability layers and a documented fallback order.
• •
  Parse-failure rate at or near 0% and field-level accuracy measured against ≥20 labeled examples.
• •
  Retries, failures, and "unknown" enum rates logged as metrics, not swallowed.
• •
  A colleague can swap the schema for a new extraction task in under 30 minutes without touching the reliability machinery.