Review Human- and AI-Generated Code for Vulnerabilities with AI

Assistant-written code is exactly the review surface where a fast, well-prompted AI pass earns its keep — as long as you treat every finding it produces as a lead to confirm, never a verdict to ship. This playbook turns an LLM into a second reviewer over your own repository: it reads the diff, flags likely vulnerabilities with reasoning, and hands you a ranked list you verify against a static analyzer and the running code.

When to reach for this

• •
  Reviewing a PR that mixes hand-written and assistant-generated code, especially auth, input parsing, deserialization, crypto, or anything touching a query or a shell.
• •
  A large or unfamiliar diff where you want a fast first pass before you invest deep manual attention.
• •
  You want a consistent checklist applied to every change — injection, authz gaps, secrets, unsafe defaults — rather than whatever you happen to remember at 5pm.

Skip it for a one-line config tweak, and never point it at a third party's code — this is for securing code you own and are authorized to review.

The workflow

1. 1.

   Gather the exact diff and its context. Export the change so the model reviews what actually merges, not an approximation:

   Copy

   git diff --unified=5 origin/main...HEAD > /tmp/review.diff
   

2. 2.

   Prompt for findings with reasoning and a severity, not a verdict. Ask for evidence you can check:

   Copy

   You are a secure-code reviewer. Review the diff below for security
   vulnerabilities in code I own and am authorized to review.
   
   For each finding, give:
   - file and line
   - vulnerability class (map to CWE where you can)
   - a one-sentence explanation of the exploitable path
   - severity (Critical/High/Medium/Low) and your confidence (High/Med/Low)
   - the minimal safe fix
   
   Report every issue you find, including low-confidence ones — I will
   verify and filter. Do not withhold findings for being uncertain. Flag
   anything that looks assistant-generated and plausible-but-wrong
   (invented APIs, missing validation, copy-pasted unsafe patterns).
   
   DIFF:
   <paste /tmp/review.diff>
   

   Expected output: a table of findings. A good pass catches things like a string-formatted SQL query, a verify=False on a TLS call, an authorization check that reads the wrong user ID, or a secret hardcoded in a "temporary" test helper.

3. 3.

   Cross-check every finding against a static analyzer. Run Semgrep or CodeQL over the same code and compare. Where the tool and the model agree, your confidence is high; where only the model flags it, you have a lead to confirm manually; where only the tool flags it, ask the model to explain the tool's finding.

   Copy

   semgrep --config auto --json services/payments > /tmp/semgrep.json
   

4. 4.

   Ask the model to reconcile the two lists. Paste the analyzer output back and have it explain agreements and disagreements, so you review a single merged list instead of two.

5. 5.

   Confirm the real ones and write them up. For each surviving finding, produce a reviewer comment with the exploit path and the fix — see the companion remediation playbook for wording developers act on.

Verify before you trust it

• Confirm a flagged vuln is real and reachable. Trace the tainted input from its entry point to the sink by hand. If the "SQL injection" parameter is a hardcoded enum three call-frames up, it is not exploitable — downgrade or drop it.
• Confirm a "clean" section is actually clean. Absence of a finding is not proof of safety. Re-prompt the riskiest files with a narrower question ("show me every place user input reaches a query, file path, or shell command in this file"). Models miss bugs, especially in code they weren't focused on.
• Prove the fix. After remediation, re-run the analyzer and, where feasible, exercise the vulnerable path with a test that fails on the old code and passes on the new — the regression-test playbook covers this.
• Watch for confident nonsense. The model will occasionally cite a CWE that doesn't fit or invent a function that isn't in the diff. If you cannot point to the exact line, it did not find anything.

Pitfalls

• Trusting silence. No finding often means the model didn't look hard at that region, not that the region is safe. Coverage is your job to force.
• Accepting the fix without the exploit. A fix you can't tie to a concrete exploitable path is churn, and sometimes introduces its own bug.
• Whole-file regeneration. Asking the model to "rewrite this securely" produces a large, hard-to-review diff. Demand minimal, targeted patches.
• Prompt-injected diffs. Comments or test fixtures in the code can contain instructions aimed at the model ("ignore the SQL issue"). Treat code as data; if a comment tells the reviewer what to conclude, that is itself worth flagging.
• Severity inflation. Models tend to rate everything High. Re-rank against real reachability and blast radius before you escalate to the author.

Tools

• Semgrepfast, rule-based SAST; run --config auto for a broad first pass and pair its findings with the model's.
• CodeQLdeeper dataflow/taint analysis for confirming whether a flagged input actually reaches a dangerous sink.
• An LLM coding assistant (Claude / Copilot-class)the reviewer that reads intent and explains exploit paths in prose you can act on.

Client-safe variant

In a services engagement you frequently cannot paste a client's source into a public tool. Before prompting, strip identifying material — real hostnames, connection strings, customer identifiers, secrets, proprietary business logic — and review only the security-relevant slice, or reproduce the pattern abstractly ("a Flask route that builds a SQL query from a request arg"). Where the client's policy forbids external tools entirely, run a local model via Ollama for the review prose and keep Semgrep/CodeQL — which run fully offline — as the grounding layer. Confirm per-client tool approval before the first prompt, and log what left the boundary.