Lab 06 · API Suite from OpenAPI with an Assertion Audit

Rahul's team generates API tests from their OpenAPI spec on every sprint and the suite is always green. Green against what, though? A generated suite inherits the spec's optimism: it asserts what should happen on the happy path and rarely pins the contract hard enough to notice a server drifting away from it. In this lab you author the spec and the server yourself — with three deliberate divergences between them — generate a suite with AI, and measure its catch rate. The gap between "suite passes" and "server conforms" is the whole lesson, and Schemathesis joins as the property-based referee that attacks the contract for you.

Time, tier & cost

~90 minutes hands-on · Core tier (maps to module M2 and playbook PB-10). Cost: $0 (local server, free AI tier, pipx install schemathesis).

Setup

mkdir -p qa-lab-06 && cd qa-lab-06 && npm init -y
npm install express@4 && npm install --save-dev @playwright/test
pipx install schemathesis   # or: pip install schemathesis

openapi.yaml — the contract (this is the truth; the server will lie):

openapi: 3.0.3
info: { title: Bookmarks API, version: 1.0.0 }
paths:
  /bookmarks:
    get:
      responses:
        "200":
          description: list
          content:
            application/json:
              schema:
                type: array
                items: { $ref: "#/components/schemas/Bookmark" }
    post:
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              required: [url, title]
              properties:
                url: { type: string, format: uri }
                title: { type: string, maxLength: 120 }
      responses:
        "201": { description: created }
        "400": { description: validation error }
  /bookmarks/{id}:
    delete:
      security: [{ bearerAuth: [] }]
      parameters:
        - { name: id, in: path, required: true, schema: { type: integer } }
      responses:
        "204": { description: deleted }
        "401": { description: missing/invalid token }
        "404": { description: not found }
components:
  securitySchemes:
    bearerAuth: { type: http, scheme: bearer }
  schemas:
    Bookmark:
      type: object
      required: [id, url, title, createdAt]
      properties:
        id: { type: integer }
        url: { type: string }
        title: { type: string }
        createdAt: { type: string, format: date-time }

server.js — the implementation. Type it exactly; the divergences are planted:

const express = require("express");
const app = express();
app.use(express.json());
let bookmarks = [];
let nextId = 1;
app.get("/bookmarks", (req, res) => {
  res.json(bookmarks.map(({ id, url, title, created }) =>
    ({ id, url, title, created })));
});
app.post("/bookmarks", (req, res) => {
  const { url, title } = req.body || {};
  if (!url || !title) return res.status(400).json({ error: "url and title required" });
  const bm = { id: nextId++, url, title, created: new Date().toISOString() };
  bookmarks.push(bm);
  res.status(200).json(bm);
});
app.delete("/bookmarks/:id", (req, res) => {
  const idx = bookmarks.findIndex((b) => b.id === Number(req.params.id));
  if (idx === -1) return res.status(404).end();
  bookmarks.splice(idx, 1);
  res.status(204).end();
});
app.listen(4000, () => console.log("Bookmarks API on :4000"));

Run node server.js. Create audit.md for your findings.

Steps

1. 1Generate the suite from the spec only. Paste openapi.yaml (not the server code) into Claude/ChatGPT: "Generate a Playwright API test suite (@playwright/test with request fixture) for this OpenAPI spec, base URL http://localhost:4000. Cover happy paths, validation errors, auth, and not-found for every operation. Assert status codes and response schemas." Save to tests/api.spec.js untouched. Skim it and predict in audit.md: which of the three bug classes named in this lab's lede do you think this suite will catch?
2. 2Run the generated suite as-is.npx playwright test. Record exactly which tests pass and fail. Typical outcome: most pass; perhaps one fails. Every passing test against this deliberately non-conformant server is a data point about assertion weakness — log the pass/fail table in audit.md.
3. 3Hunt divergence #1 — status codes. Compare the POST behavior against the spec by hand: curl -i -X POST localhost:4000/bookmarks -H 'content-type: application/json' -d '{"url":"https://x.dev","title":"x"}'. The spec says 201; note what the server returns and whether your generated suite asserted the exact code or something looser (ok(), status < 400). Record the finding and the exact assertion that let it slip (or caught it).
4. 4Hunt divergence #2 — contract drift.curl -s localhost:4000/bookmarks | python3 -m json.tool after creating one bookmark. Diff the object's fields against the Bookmark schema (required: [id, url, title, createdAt]). Two problems hide here — a renamed field and a missing required field, which are really one root cause. Check whether any generated test validated response shape at all, or only status codes. Record it.
5. 5Hunt divergence #3 — the missing auth check. The spec marks DELETE with bearerAuth and a 401 response. Test it: curl -i -X DELETE localhost:4000/bookmarks/1 with no token. Record what happens. This is the highest-severity class in the lab — a security control that exists only in documentation. Note whether the AI generated a "DELETE without token → 401" negative at all; suites generated from specs frequently test auth with a token but never without.
6. 6Let Schemathesis attack the contract. Run schemathesis run openapi.yaml --url http://localhost:4000 --checks all and save the output. Compare its findings to yours: it should flag the undocumented 200-vs-201 and schema non-conformance on GET; observe whether it flags the missing auth enforcement (negative-auth is a check it can make when the scheme is declared — verify rather than assume). Add a row to audit.md: found by AI suite / found by hand / found by Schemathesis, for each divergence.
7. 7Strengthen the suite. Now edit tests/api.spec.js (AI may help, but you specify the assertions): exact status-code equality everywhere; a schema assertion on every JSON response — write a tiny assertBookmark(obj) helper that checks all four required fields and their types, so createdAt-vs-created fails loudly; the three auth negatives (no token → 401, malformed token → 401, valid delete → 204); boundary negatives from the spec (title of 121 chars → 400, non-URI url → 400, non-integer id → 404 or 400 — decide and document which the contract implies). Run the suite: it must now fail on exactly the planted divergences.
8. 8Fix the server, prove the loop. Correct all three divergences in server.js (201 on create; rename created → createdAt everywhere; enforce a bearer token on DELETE — any non-empty Authorization: Bearer value counts for the lab). Rerun the strengthened suite (green) and Schemathesis (clean, or with only documented findings). Close audit.md with the catch-rate table: generated suite 0–1 of 3, strengthened suite 3 of 3.

Verify

• •
  audit.md contains your step-1 prediction, the raw pass/fail table, and the three divergences each attributed (AI suite / manual / Schemathesis).
• •
  The strengthened suite failed on all three divergences before the server fix and passes after.
• •
  A response-shape helper validates all four required Bookmark fields on every JSON-returning call.
• •
  Auth negatives exist: absent and malformed tokens both asserted as 401.
• •
  Schemathesis output saved from both the broken and fixed server runs.

Self-grading rubric (pass = 7/10):