Test Structured-Output Contracts — Schema Validity Under Repeated Sampling

Tier: Core · Workflow area: Testing AI systems — foundations

You will define the contract in Pydantic/JSON Schema, measure validity rates across strata with repeated sampling, verify the repair pipeline actually repairs, and fault-inject malformed outputs into the consumer — because the contract test is not "does the model comply" but "does the system hold when it doesn't."

When to use

Prerequisites

• •
  The contract as code: a Pydantic model or JSON Schema file in the repo, versioned — the same artifact production uses, not a test-only copy.
• •
  API access to the feature with production config; Promptfoo or a Python harness for repetition.
• •
  A representative input slice from the golden dataset, including hard strata (long inputs, weird characters, ambiguous cases).

The recipe

1. 1.

   Make the contract strict and singular. One schema artifact, shared by production and tests:

   Copy

   from pydantic import BaseModel, Field
   from typing import Literal
   
   class TicketExtraction(BaseModel, extra="forbid"):
       category: Literal["billing", "shipping", "product", "account", "other"]
       priority: Literal["p1", "p2", "p3"]
       summary: str = Field(max_length=400)
       order_ids: list[str] = Field(default_factory=list)
   

   extra="forbid" matters: models pad objects with unrequested fields, and tolerant parsing lets schema drift accumulate invisibly until something downstream depends on a field the model sometimes forgets.

2. 2.

   Measure validity as a rate, stratified, under repeated sampling.

   Copy

   from collections import Counter
   results = Counter()
   for case in golden_slice:                    # include the hard strata
       for _ in range(20):
           raw = extract(case.input)            # production code path
           try:
               TicketExtraction.model_validate_json(raw)
               results[case.stratum, "valid"] += 1
           except Exception as e:
               results[case.stratum, type(e).__name__] += 1
               log_sample(case.id, raw, e)      # keep the malformed bodies!
   

   Copy

   happy_path   : 400/400 valid (1.000)
   long_input   : 187/200 valid (0.935)  — 9 truncated JSON, 4 extra field
   edge_format  : 278/300 valid (0.927)  — 14 markdown-fenced, 8 prose preamble
   

   Two lessons appear in every first run: validity is input-dependent (aggregate rates hide the weak stratum), and failures cluster into a small taxonomy — truncation, code-fence wrapping, prose preamble, invented fields, wrong enum casing. Keep the malformed samples; steps 4–5 feed on them.

3. 3.

   The same rig in Promptfoo, if you prefer config:

   Copy

   defaultTest:
     options: { repeat: 20 }
     assert:
       - type: is-json
         value: file://schemas/ticket_extraction.json
   tests:
     - vars: { ticket: file://golden/long-input-01.txt }
       threshold: 0.95
   

   is-json with a schema file gives per-case validity rates and CI thresholds with no harness code — right choice when the prompt-regression gate from the previous playbook already exists.

4. 4.

   Test the repair pipeline as its own unit. Production should already have one (strip fences → retry with error feedback → fallback); test each stage against the real malformed samples collected in step 2:

   Copy

   @pytest.mark.parametrize("raw", load_malformed_samples())
   def test_repair_pipeline(raw):
       result = repair_and_validate(raw, TicketExtraction)
       assert result.ok or result.fell_back    # never a raised exception
   

   Then measure the composite rate: raw validity 93.5% + one feedback-retry may compose to 99.8%. The composite, not the raw rate, is your real contract number — and both belong in the release criteria (raw rate guards cost and latency; composite guards correctness).

5. 5.

   Fault-inject at the consumer boundary — the half everyone skips. The contract test is bidirectional: prove the consumer survives contract breaches. Have an assistant generate the attack set from your schema:

   Copy

   Given this Pydantic model, generate 25 malformed variants a language
   model might realistically emit: truncated mid-string, markdown-fenced,
   single-quoted keys, wrong enum casing ("Billing"), extra fields,
   null where a list is required, valid JSON of the WRONG shape,
   prose apology followed by valid JSON. Output as a JSON array of strings.
   

   Feed each variant into the consumer in a test. Required behavior: reject cleanly, route to fallback, log with the sample attached — never a stack trace in a user path, never a silent partial write. A consumer that only ever met valid outputs in testing has an untested error path in production by definition.

6. 6.

   Gate and monitor on rates. CI: per-stratum raw validity ≥ its threshold (e.g. 0.95 hard strata, 0.99 happy path), composite ≥ 0.999, and the fault-injection suite green. Production: emit a schema_valid boolean metric per call and alert on rate drops — a provider-side model update can move validity overnight with zero deploys on your side, and this metric is how you find out before your users do.

Variations

• Provider structured-output modes: JSON/tool-call modes typically guarantee syntactic JSON, not your semantic constraints (enum membership, lengths, extra="forbid"). Rerun step 2 after enabling — validity rises, rarely to 1.0 for strict schemas; the rig stays.
• Streaming consumers: truncation dominates. Add explicit tests for parse-on-incomplete-stream and assert that partials are never committed downstream.
• Tool-calling agents: each tool's argument schema is a separate contract; run per-tool validity rates — one flaky tool schema can dominate an agent's end-to-end pass^k.

Quality check

• •
  One schema artifact shared by prod and tests; its version is stamped into eval results.
• •
  Validity measured per stratum at N≥20 with malformed samples archived; raw and composite rates both recorded.
• •
  Repair pipeline unit-tested against real (not only synthetic) malformed samples.
• •
  Fault-injection suite covers the full taxonomy including wrong-shape-but-valid JSON — the nastiest class, because json.loads succeeds.
• •
  Production emits and alerts on the validity metric; the alert has fired in a drill.

Pitfalls

• Testing with json.loads alone. Syntactic validity without schema validation misses wrong shapes, wrong enums, and extra fields — most of the taxonomy.
• Retry-blind cost math. A 93% raw rate with feedback-retry means ~7% of calls cost double latency and tokens. If the raw rate slides while composite stays green, nobody notices until the bill or the p95 does. Track both.
• Tolerant parsing as a fix. Loosening the schema (extra="allow", optional-everything) converts contract failures into silent data-quality failures. Repair explicitly; never relax the contract to make tests pass.
• One aggregate threshold. 0.98 overall can be 1.0 on happy path and 0.85 on the long-input stratum your biggest customer lives in. Strata or it didn't happen.