Generate Security Regression Tests That Lock a Fixed Bug Shut

A security regression test is the guard: a test that fails against the vulnerable code and passes against the patched code, wired into CI so any future reintroduction breaks the build. AI is excellent at drafting these — it can turn an exploit path into a concrete test case quickly — but the test is only trustworthy once you've proven it actually fails on the old code. A green test that would stay green even if the bug returned is worse than no test, because it manufactures false confidence.

When to reach for this

• •
  You just fixed a confirmed vulnerability and want to prevent its silent reintroduction.
• •
  You're building out a security regression suite — encoding known-bad inputs so the CI pipeline enforces them forever.
• •
  A vulnerability class recurs in your codebase (the same injection pattern in three services) and you want parameterized tests covering the family.

The workflow

1. 1.

   Start from the confirmed exploit, not the fix. You need the exact input that triggered the vulnerability — the payload, the request, the state — because that is what the test must exercise.

2. 2.

   Prompt for a test that encodes the exploit as an assertion:

   Copy

   You are writing a security regression test for a vulnerability I fixed in
   my own code. Write a test that:
   - exercises the exact exploit path (the malicious input that triggered it)
   - asserts the SECURE behavior (input rejected / query parameterized /
     access denied), so it FAILS on the vulnerable version and PASSES on the
     fixed version
   - is deterministic and needs no external services
   Name the CWE in a comment. Use <pytest / the repo's framework>.
   
   VULNERABILITY: path traversal (CWE-22) in /download?file=
   EXPLOIT INPUT: file=../../etc/passwd
   FIXED BEHAVIOR: request returns 403, no file outside the allowed dir
   FIXED CODE:
   <paste the patched handler>
   

   Expected output: a self-contained test that sends the traversal payload and asserts a 403 (or that no out-of-tree path is read) — an assertion that is false on the old code and true on the new.

3. 3.

   Prove it fails on the vulnerable code — the non-negotiable step. Check out the pre-fix commit (or revert the patch locally) and run the new test. It must fail. If it passes against the vulnerable code, the test is not testing the vulnerability — send it back:

   Copy

   git stash            # set aside the fix
   pytest tests/test_download_traversal.py   # MUST fail here
   git stash pop        # restore the fix
   pytest tests/test_download_traversal.py   # MUST pass here
   

4. 4.

   Parameterize for the family where the bug recurs. If the same class appears in several places, ask the model to expand the payload set (encoded traversal %2e%2e%2f, absolute paths, null bytes) so one test covers the variants an attacker would actually try.

5. 5.

   Wire it into CI as a gate. Add the test to the security suite and make its failure block merge — that is what turns a one-time fix into a durable control.

Verify before you trust it

• Run the red-then-green proof, every time. This is the verification. No red on the vulnerable code means the test is theater. Automate it if you can (run the test against the reverted patch in CI once, to certify it).
• Confirm the test asserts security, not incidental behavior. A test that passes because the endpoint happens to 404 for an unrelated reason will keep passing after the bug returns via a different route. The assertion must be about the security property.
• Confirm determinism. Re-run the test several times and in CI. A flaky security test gets quarantined, which quietly disables your guard.
• Check the payload set covers real attacker variants. The model's first draft often tests one canonical payload; attackers use encodings and edge cases. Expand until the family is covered.

Pitfalls

• Skipping the red proof. The cardinal sin. A test you never saw fail on the vulnerable code is unverified and may guard nothing.
• Asserting the wrong thing. Passing for an incidental reason (wrong status code, unrelated exception) means the test survives the bug's return. Assert the security property directly.
• Environmental dependence. A test that needs a live database or network is flaky and gets disabled. Keep it hermetic.
• Single-payload coverage. One traversal string tests one path; the encoded and absolute variants are how the bug comes back. Parameterize.
• Not gating CI. A regression test that runs but doesn't block merge is documentation, not a control. Make failure fail the build.

Tools

• An LLM (Claude-class)turns an exploit path into a concrete, framework-idiomatic test and expands the payload family.
• Semgrepas a complementary guard: a custom rule that flags the vulnerable pattern's return catches reintroductions the unit test's specific payload might miss.
• pytest (or the repo's test framework)runs the red-then-green proof and hosts the test in the CI security suite.

Client-safe variant

Regression tests embed the client's code and a working exploit payload against their system. Before prompting, redact real paths, hosts, and credentials, and reduce the handler to the security-relevant logic. If external tools are barred, run a local model for the test drafting — the test runner and Semgrep execute entirely offline. Never paste a live production exploit into a public tool; reproduce the pattern against a local sandbox instead. Keep tests in the client's own repository and CI, and confirm per-client tool approval before prompting.