Encode Team Standards into AI Review Instructions (and Tune the Noise Out)

Tier: Core · Workflow area: Code Review

This playbook covers writing the instructions, seeding test PRs to measure signal-vs-noise, and iterating until the bot earns its place in the pipeline.

When to use

• •
  Your team has adopted (or is trialing) Copilot code review, CodeRabbit, or a Roast-style self-hosted reviewer.
• •
  Review comments keep repeating the same 5–10 team-specific rules humans are tired of writing.
• •
  Developers have started ignoring the AI reviewer — the classic symptom of noise domination.

Prerequisites

• •
  Repo admin rights to add instructions files and configure the review tool.
• •
  A written (or extractable) list of team standards. If they only exist in review history: gh pr list --state merged --limit 50 then mine comments for repeated feedback.
• •
  Two sacrificial test PRs you can open and close for calibration.

The workflow

1. 1.

   Extract 5–8 enforceable standards. Good instructions are checkable from a diff; aspirations are not. Convert each recurring review comment into a rule with a detection condition:

   • •
     Weak: "Write good error handling."
   • •
     Strong: "Every except/catch must either re-raise, return an explicit error value, or log with logger.exception including context — flag bare except: pass and log-and-continue as blocking."
2. 2.

   Write the instructions file. For GitHub Copilot code review, create .github/copilot-instructions.md (as of the July 2026 changelog, instructions are read from the head branch, so PRs can ship rule updates with the code they apply to):

   Copy

   # Code review instructions
   
   Review priorities, highest first:
   1. FLAG (blocking): raw SQL string interpolation — all queries must use
      parameterized bindings or the query builder in `app/db/`.
   2. FLAG (blocking): new public API endpoints without a rate-limit decorator
      and an entry in `openapi.yaml`.
   3. FLAG: any `except`/`catch` that swallows errors (see error policy).
   4. FLAG: new env var read without a default + entry in `config/schema.py`.
   5. FLAG: tests asserting on log output or sleep()-based timing.
   
   Do NOT comment on: formatting (black/prettier owns it), import order,
   docstring style, variable naming unless misleading, anything in
   `migrations/` or generated files.
   

   The "do NOT comment on" block is as important as the rules — it is your noise valve.

3. 3.

   Seed a calibration PR. Open a test PR containing 5 deliberate violations (one per rule) buried in ~200 lines of otherwise-clean change, plus 3 "trap" patterns that look flaggable but are explicitly excluded (e.g., a formatting quirk, a migrations file). Request AI review.

4. 4.

   Score it like a classifier.

   • •
     Recall: violations caught / 5 seeded. Below 4/5 → the rule wording is too abstract; add a concrete bad-example snippet to the instructions.
   • •
     Precision: real findings / total comments. Below ~70% → expand the do-not-comment list with the exact noise categories observed.
   • •
     Traps flagged → your exclusions are not being honored; state them more literally.
5. 5.

   Iterate to threshold, then run a clean-PR control. Repeat with the second test PR (different seedings) until you hold ≥4/5 recall with ≤30% noise. Then run the reviewer on a genuinely clean PR — it should produce at most 1–2 comments. A bot that always finds something trains people to ignore it.

6. 6.

   Ship with a feedback loop. Merge the instructions file; tell the team the rule set is a PR-reviewable artifact — anyone can propose a rule change like any code change. Re-score quarterly or when the tool ships model updates, using the same seeded-PR method.

Tool variants

• GitHub Copilot code review:.github/copilot-instructions.md plus per-path instruction files; org policy can enforce firewalled review for private code.
• CodeRabbit:.coderabbit.yaml with path_instructions for per-directory rules and tone_instructions; free on public repos makes it good practice ground.
• Self-hosted (Shopify Roast pattern / custom): a scripted LLM pass over the diff in CI with your rules in the prompt — full control of model and data path; pair with Semgrep so deterministic rules stay deterministic.

Verification

• •
  Calibration recall ≥4/5 seeded violations across two different seeded PRs.
• •
  Noise rate ≤30% of comments, and the clean-PR control produced ≤2 comments.
• •
  Instructions file lives in the repo, versioned, with an owner and a review path for changes.
• •
  One month in: developers still act on the bot's blocking findings (spot-check 5 recent PRs for responses vs. ignores).

Pitfalls

• Encoding taste instead of standards. Rules a human reviewer would not consistently enforce become noise in bot form. If the team argues about it, it is not ready for the bot.
• Anything a linter can check does not belong in the LLM reviewer. Push formatting, import order, and mechanical patterns to ESLint/Ruff/Semgrep — deterministic tools do not hallucinate and cost nothing.
• Measuring only recall. A reviewer that catches everything by flagging everything has destroyed its own credibility; precision is what keeps humans listening.
• Forgetting model drift. Tool vendors swap underlying models; your tuned behavior shifts silently. Re-run the seeded PRs after major tool updates.

Restricted-environment variant

• •
  For client repos where diffs must not leave approved infrastructure: use the review tool's enterprise/firewalled mode (Copilot code review with firewall settings, or client-tenant hosted models via a Roast-style script hitting Bedrock/Azure OpenAI inside the client cloud).
• •
  Fully restricted route: run the same instructions as a local-model CI pass (Ollama on a build agent) — quality drops on subtle rules, so shift more rules into Semgrep (free CE) and keep only the judgment-heavy rules on the LLM.
• •
  The seeded-PR calibration method is tool-agnostic and works identically in every environment — it is the part to standardize across client accounts.

Success criteria

• •
  The bot enforces the team's top rules with measured recall/precision, and repeat nit comments from humans drop visibly.
• •
  New team members learn the standards from bot feedback within their first PRs — the instructions file doubles as onboarding.
• •
  Rule changes go through PR review, and quarterly re-calibration is on the team calendar.