Lab 03 · Bulletproof Structured Outputs

Prompt-only JSON extraction shows double-digit failure rates in production; the layers are how real data products close the gap.

Time & tier

~2 hours hands-on · Core tier · API spend ≤ $0.50 (240–360 cheap-tier calls).

Setup

Python 3.10+, pydantic>=2, openai SDK (any OpenAI-compatible endpoint: OpenAI, Gemini-compatible, Groq). Generate the corpus — deterministic (seed 7), stdlib only, with ground truth for scoring:

"""make_order_emails.py — 120 messy order-confirmation emails + ground truth.
Writes emails.jsonl + truth.jsonl."""
import json
import random

rng = random.Random(7)
SKUS = ["KB-MECH-87", "MS-ERGO-2", "MON-27-4K", "DOCK-USB4", "CAM-1080P", "HS-ANC-X"]
CCY = [("INR", "Rs.", 83.0), ("USD", "$", 1.0), ("EUR", "EUR ", 0.92)]
DATE_STYLES = ["{y}-{m:02d}-{d:02d}", "{d:02d}/{m:02d}/{y}", "{month} {d}, {y}"]
MONTHS = ["January", "February", "March", "April", "May", "June", "July",
          "August", "September", "October", "November", "December"]

emails, truth = [], []
for i in range(120):
    oid = f"ORD-{rng.randint(100000, 999999)}"
    cust = f"user{rng.randint(100, 999)}@{rng.choice(['gmail.com', 'yahoo.in', 'outlook.com'])}"
    ccy, sym, _ = CCY[rng.randrange(3)]
    items = [{"sku": rng.choice(SKUS), "qty": rng.randint(1, 4)}
             for _ in range(rng.randint(1, 3))]
    total = round(rng.uniform(40, 900) * (83 if ccy == "INR" else 1), 2)
    y, m, d = 2026, rng.randint(1, 12), rng.randint(1, 28)
    date_iso = f"{y}-{m:02d}-{d:02d}"
    style = rng.randrange(3)
    date_txt = DATE_STYLES[style].format(y=y, m=m, d=d, month=MONTHS[m - 1])

    lines = [f"Subject: Your order {oid} is confirmed!", "",
             f"Hi, thanks for shopping with us. Order ref: {oid}."]
    for it in items:
        lines.append(f"  - {it['qty']} x item {it['sku']}")
    r = rng.random()
    if r < 0.25:
        lines.append(f"Grand total comes to {sym}{total:,.2f} (incl. all taxes).")
    elif r < 0.45:
        lines.append(f"Total: {total:,.2f} {ccy}")
    else:
        lines.append(f"Amount charged: {sym}{total}")
    if rng.random() < 0.30:
        cust_txt = cust.replace("@", " [at] ").replace(".", " dot ")
        lines.append(f"Receipt sent to {cust_txt}")
    else:
        lines.append(f"Receipt sent to {cust}")
    lines.append(f"Expected delivery: {date_txt}")
    if rng.random() < 0.15:
        lines.append(f"(Related return ORD-{rng.randint(100000, 999999)} was closed.)")

    emails.append({"id": i, "text": "\n".join(lines)})
    truth.append({"id": i, "order_id": oid, "customer_email": cust,
                  "total_amount": total, "currency": ccy,
                  "items": items, "delivery_date": date_iso})

with open("emails.jsonl", "w") as f:
    f.writelines(json.dumps(e) + "\n" for e in emails)
with open("truth.jsonl", "w") as f:
    f.writelines(json.dumps(t) + "\n" for t in truth)
print("wrote 120 emails + ground truth")

Keep truth.jsonl closed until scoring — it is the answer key.

Guided steps

1. 1.
   Write the schema as the contract, with constraints strict enough to catch real damage — patterns, ranges, enums, and a custom validator:

import json
import re
from typing import List, Literal
from pydantic import BaseModel, Field, field_validator

class Item(BaseModel):
    sku: str = Field(pattern=r"^[A-Z0-9]+(-[A-Z0-9]+)+$")
    qty: int = Field(ge=1, le=10)

class Order(BaseModel):
    order_id: str = Field(pattern=r"^ORD-\d{6}$")
    customer_email: str
    total_amount: float = Field(gt=0)
    currency: Literal["INR", "USD", "EUR"]
    items: List[Item] = Field(min_length=1)
    delivery_date: str = Field(pattern=r"^\d{4}-\d{2}-\d{2}$")

    @field_validator("customer_email")
    @classmethod
    def email_shape(cls, v: str) -> str:
        if not re.fullmatch(r"[^@\s]+@[^@\s]+\.[^@\s]+", v):
            raise ValueError("not a de-obfuscated email address")
        return v

2. 1Layer 0 — naive baseline. One prompt asking for JSON, json.loads on the raw reply, no schema. Run all 120; record parse-rate and (for parses) validity against Order. This is your "before" number.
3. 2Layer 1 + 2 — validate, then retry with the error fed back:

def extract_json(text):
    m = re.search(r"\{.*\}", text, re.DOTALL)
    if not m:
        raise ValueError("no JSON object found")
    return json.loads(m.group(0))

def layered_extract(email_text, call_llm, max_retries=2):
    prompt = (
        "Extract the order as JSON with keys order_id, customer_email, "
        "total_amount (number, no separators), currency (INR|USD|EUR), "
        "items (list of {sku, qty}), delivery_date (YYYY-MM-DD). "
        "De-obfuscate emails written as 'x [at] y dot com'. If two ORD- codes "
        "appear, extract the order being confirmed, not the closed return.\n\n"
        + email_text)
    last_err = None
    for attempt in range(max_retries + 1):
        raw = call_llm(prompt if attempt == 0
                       else prompt + f"\n\nYour previous output failed validation: "
                                     f"{last_err}\nReturn corrected JSON only.")
        try:
            return Order.model_validate(extract_json(raw)), attempt
        except Exception as e:
            last_err = str(e)[:400]
    return None, max_retries + 1

call_llm is any function prompt -> str (temperature 0). This is the Instructor pattern hand-rolled so you can see it; swapping in instructor.from_openai(...) afterwards is a one-liner and a good extension. 4. Layer 3 — constrained decoding fallback. For records still failing after retries, call your provider's native structured-output mode (response_format with a JSON schema, or a forced tool call) as the fallback path. Providers guarantee syntactic validity here; your Pydantic constraints still arbitrate semantic validity (a syntactically perfect "currency": "GBP" must still fail). 5. Measure everything. For each layer cumulatively, over all 120 emails: schema-validity rate, retry count, and field accuracy vs truth.jsonl — exact match for order_id, customer_email, currency, delivery_date; abs(diff) < 0.01 for total_amount; set-equality for items. Also track tokens so you can price the reliability: retries cost money, and the report should say how much. 6. Wrap it as a service (optional, +30 min): a FastAPI POST /extract endpoint returning the validated Order or a structured error. The measurement harness then doubles as your integration test.

Verification

• •
  Layer-by-layer table shows monotonic improvement, e.g. Layer 0 typically 75–92% valid on this corpus with a cheap model; Layer 2 ≥97%; Layer 3 ≥99%.
• •
  The decoy trap: among emails containing a second ORD- code (~15%), order_id accuracy ≥90% — if not, your prompt's decoy instruction is being ignored and needs strengthening.
• •
  Obfuscated emails (~30% of corpus) are de-obfuscated to pass the validator, not dropped.
• •
  total_amount accuracy ≥95% including comma-formatted amounts (the validator rejecting "1,234.50" as a string is the designed failure your retry layer must fix).

Completion artifact

reliability-report.md: the layer table (validity, accuracy, retries, cost), one paragraph on which messiness patterns caused which failures, and your harness code. Self-grading rubric (pass = 8/10):

• Layering (0–3): three distinct mechanisms measured cumulatively, not one blended run.
• Target (0–2): ≥99% schema-validity at the final layer; ≥95% total-amount accuracy.
• Semantic honesty (0–2): report separates "valid JSON" from "correct values" — they are different curves.
• Cost awareness (0–2): $/1k extractions at each layer computed from measured tokens.
• Reusability (0–1): harness runs end-to-end with one command.

Stuck?

• Validity stuck near 90% after retries. Print the top five validation errors; usually one systematic pattern (comma numbers, date style 3) dominates — fix it in the prompt, not with more retries.
• Provider has no structured-output mode. Use a forced tool/function call, or an open-weights path with Outlines for grammar-constrained decoding; the measurement protocol is unchanged.
• Truth mismatches on items. Duplicate SKUs in one order are legitimate; compare as multisets.

Client-safe variant

The pattern transfers to client document extraction unchanged, but generate a synthetic corpus in the client's domain (as done here) for development, and touch real client records only inside the approved tenancy. Ship the layer table with every extraction service you deliver — clients accept "99.2% valid, 96% accurate, here is the failure taxonomy" far better than a demo that worked once.

Answer key (don't read before attempting)

Planted messiness (all seeded): 25% of totals rendered with currency symbol inside prose, 20% with thousands separators and trailing currency code, remainder bare; 30% of customer emails obfuscated ([at]/dot); delivery dates in three styles (ISO, DD/MM/YYYY, "Month D, YYYY") — two require normalization; ~15% of emails carry a decoy closed-return ORD- code; items may repeat the same SKU. The intended layer effects: comma-formatted totals and non-ISO dates break Layer 0/1 and are recovered by the retry with error feedback; stubborn prose-total and decoy cases are what Layer 3 plus the decoy prompt line exist for.