Your OpenAPI or proto file is the most under-exploited test asset you own: a machine-readable contract that AI can turn into integration tests covering not just the 200-path, but the cases that actually page you — timeouts, 429s, malformed payloads, missing fields, and the client that retries a non-idempotent POST.
Tier: Core · Workflow area: Testing & Quality
This playbook combines spec-driven generation (schemathesis for breadth) with AI-authored failure-mode tests (for depth), so "the spec" and "the service" stop being two different fictions.
Skip for internal function boundaries within one deployable — contract ceremony there is overhead; unit tests suffice.
openapi.yaml, .proto) — current, not the one from two quarters ago. If unsure, regenerate from code annotations first.Let the property tool take the breadth. Run schemathesis before writing anything by hand: schemathesis run --checks all --stateful=links openapi.yaml --base-url http://localhost:8000. It fuzzes every endpoint against its own schema and finds the cheap lies (undocumented 500s, response bodies violating their declared schema). Fix or file those first.
Have AI mine the spec for failure scenarios. Paste the spec:
Here is our OpenAPI spec (below). For POST /v1/payments and GET /v1/payments/{id}:
1. List the failure-mode scenarios a consumer must survive: every documented
4xx/5xx, plus undocumented realities — 429 with/without Retry-After,
connection timeout mid-request, response truncation, duplicate submission.
2. Flag contract smells: fields nullable in the schema but required in prose,
enums missing an "unknown" escape hatch, missing idempotency-key on POST.
3. Rank scenarios by production likelihood x blast radius.
The contract-smell list alone routinely pays for the session — fix the spec before testing against it.
Generate the failure-mode test suite. For the top-ranked scenarios:
Write pytest integration tests using httpx against http://localhost:8000 for:
- 429 handling: server returns 429 + Retry-After: 2; assert client retries
after the interval and succeeds; assert no retry storm (max 3 attempts).
- Idempotency: POST /v1/payments twice with the same Idempotency-Key header;
assert single payment created, second response replays the first.
- Malformed downstream: stub the FX-rate dependency to return truncated JSON;
assert we return 502 with our documented error envelope, not a raw stack trace.
Use respx to stub downstream; use the real service for our own endpoints.
Assert on the documented error envelope schema, not on prose messages.
Pin the error envelope. Ask the AI to generate a JSON-schema for your documented error shape and validate every non-2xx response body in the suite against it. Inconsistent error bodies are the most common silent contract break.
Wire it as a gate. Contract job in CI: schemathesis run + the failure-mode suite, executed against the built container. Breaking either fails the PR — which is the whole point: the spec is now enforced, not aspirational.
(Consumer-driven option) Where a specific consumer team matters more than the open contract, have the AI draft Pact interactions from that consumer's actual call sites, then verify the provider against the pact in CI.
Know the division of labor — schemathesis derives breadth mechanically from the schema; the AI adds semantic failure cases (idempotency, retry behavior, degraded dependencies) that no schema encodes. Interviewers probe whether you can name what each layer cannot do.
buf breaking cheaply; AI fills in behavioral tests over the generated stubs.assert response.status_code < 500 is contract theater — pin the specific code the spec documents.API specs are frequently NDA'd — endpoint names reveal product roadmap. Sanitize before external prompting: keep structure and types, neutralize path/field names (/v1/payments → /v1/resources), then map back locally. Better: run the whole flow in the client's sanctioned tenant. schemathesis, Pact, and buf are all local/OSS and need no external calls at all — only the AI-authored failure-scenario step needs a model, and a local Qwen-Coder-class model handles it acceptably when you supply the scenario list yourself from step 2's checklist categories.