Generate API Test Suites from OpenAPI — Then Attack Your Own Assertions

Tier: Core · Workflow area: Automation engineering

This playbook builds an API suite from a spec in two movements — generation with explicit assertion depth, then adversarial self-audit with Schemathesis and targeted assert-weakening checks. The suite that survives its own attack is the one allowed to guard the API.

When to use

• A service with an OpenAPI/GraphQL schema and thin test coveragethe highest-leverage generation target in API QA.
• Contract-drift anxietyconsumers keep breaking because responses changed shape and nothing failed.
• Third-party or internal API onboardingyou need a verification net around someone else's service before you depend on it.

When NOT to use:

• The spec is fiction. If the schema doesn't match the deployed API, fix or regenerate the spec first — tests generated from fiction institutionalize it.
• One endpoint, one checka health-check probe doesn't need this machinery.

Prerequisites

• The OpenAPI/GraphQL spec file for the target service, and a staging environment you may hammer with fuzzed requests — confirm this before running Schemathesis.
• Schemathesis installed (pip install schemathesis) plus your HTTP test stack (pytest + httpx, or Postman/Bruno).
• Credentials with test-data rightsthe suite creates and deletes its own records via the API.
• An approved assistant for spec content, per data policy.

The recipe

1. 1.

   Validate the spec itself:

   Copy

   npx @redocly/cli lint openapi.yaml
   

   Broken or vague specs (bare type: object, missing 4xx responses) cap everything downstream — tighten the worst offenders before generating.

2. 2.

   Generate the suite with assertion depth spelled out. The prompt's rules are the difference between real checks and status == 200 theater:

   Copy

   From this OpenAPI spec, generate pytest tests using httpx for the
   /orders endpoints.
   For EVERY request assert ALL of:
   - exact status code (200 vs 201 matters),
   - response validates against the spec's response schema (use
     jsonschema, load schemas from the spec, don't restate them),
   - business invariants stated in descriptions (e.g., "total equals sum
     of line items"),
   - error bodies match the documented error schema — not just the code.
   Include per endpoint: happy path, missing/invalid auth (401/403
   distinct), malformed body (422 with field-level errors), not-found,
   and boundary values from schema constraints (maxLength, minimum...).
   Base URL and token from env vars. No hardcoded IDs — create test data
   via the API itself and clean up.
   

   A generated test worth keeping looks like:

   Copy

   def test_create_order_rejects_negative_quantity(client):
       r = client.post("/orders", json=order_payload(quantity=-1))
       assert r.status_code == 422
       body = r.json()
       assert body["errors"][0]["field"] == "items[0].quantity"
       validate(body, ERROR_SCHEMA)  # documented error contract, enforced
   

3. 3.

   Run the suite, then fuzz the same surface with Schemathesis:

   Copy

   pytest tests/api/ -q
   schemathesis run openapi.yaml --base-url https://staging.example.com \
     --checks all --hypothesis-max-examples 200
   

   Schemathesis derives property-based cases from the schema — everything Schemathesis finds that your generated suite didn't is a measured gap in the generation. Triage its failures: real API bugs get filed; spec-vs-implementation mismatches get a spec fix; and each gap class gets a permanent test added to the suite.

4. 4.

   Attack your own assertions. The API-tier equivalent of mutation testing — plant response bugs and count detections. Run the suite against a wrapper that deliberately corrupts responses:

   Copy

   Write a small FastAPI reverse-proxy for my staging API that applies one
   corruption per run, selected by env var MUTATION:
   drop_field (remove a required response field), wrong_type (stringify a
   number), status_swap (200→201), extra_2xx (return 200 for a request
   that should 404), empty_error_body.
   I will run my suite against it and log which mutations go undetected.
   

   Copy

   MUTATION=drop_field uvicorn proxy:app --port 9000 &
   API_BASE=http://localhost:9000 pytest tests/api/ -q
   

   Every mutation that passes the suite is a weak assert. Feed the survivors back to the assistant to strengthen, exactly like pb-09's mutant loop.

5. 5.

   Wire into CI as a contract gate — suite plus a Schemathesis smoke (--hypothesis-max-examples 20) on every PR that touches the API or the spec, full fuzz nightly.

Make it yours

• Postman/Bruno teamshave the assistant emit a collection with test scripts (pm.test / Bruno asserts) instead of pytest; keep step 4 by running the collection through Newman against the corruption proxy.
• GraphQLgenerate operations and assertions from the SDL; replace Schemathesis with schema-aware fuzzing of nullable fields, and add depth/complexity abuse cases (the GraphQL-specific negative class the model will include if you name it).
• Consumer-driven contract shops (Pact)use this playbook for the provider-side net and let generated tests cover what consumer contracts don't: negatives, auth matrices, and boundary values that no consumer encodes.

Quality check

• Schema validation is live, not decorativetemporarily remove a field from a staging response (or use the proxy) and confirm the suite fails loudly.
• All five corruption mutations are detecteddrop_field, wrong_type, status_swap, extra_2xx, empty_error_body; survivors were strengthened and re-attacked.
• 401 vs 403 vs 404 are asserted distinctlygenerated suites blur these; auth-matrix confusion is a real vulnerability class, not pedantry.
• Schemathesis runs clean at the chosen example budget, and each earlier finding maps to a permanent regression test, not just a fixed bug.
• Tests are self-sufficientdata created and cleaned via the API; the suite passes twice in a row on the same environment.
• Spec drift fails the builda deliberate one-line spec change that no code implements should turn CI red; confirm it once.