Clean Up Leaked Secrets at Scale with AI-Drafted Remediation

Tier: Foundation · Hands-on time: ~40 minutes

The deterministic scanners are the source of truth; AI's job is the toil around them: classifying findings, drafting per-repo remediation PRs, and writing the rotation runbook for each credential type. One rule is absolute: the AI never sees the secret values, and rotation itself is executed by a human.

When to use

• •
  First estate-wide secret scan (new client onboarding, security push) produced a finding pile nobody has triaged.
• •
  Standing hygiene: secret scanning is in CI but findings accumulate faster than they are actioned.
• •
  Post-incident: a leaked credential forced rotation and you want the rest of the estate swept properly.
• •
  Bonus relevance: before adopting AI coding tools — repos full of secrets are exactly what you do not want entering assistant context windows.

Skip AI for a single confirmed live leak — that is an incident: rotate now, then come back for the sweep.

Prerequisites

• •
  Scanner output (gitleaks/TruffleHog JSON, or GitHub secret-scanning API export) with rule type, file path, and commit — with secret values masked or stripped before any AI sees the export.
• •
  Repo metadata: owner team, prod-relevance, archived-or-active.
• •
  Access to rotate (or file rotation requests for) the credential classes found: cloud keys, DB passwords, API tokens, webhook secrets.
• •
  A secrets manager to remediate INTO (Vault, cloud secrets manager, sealed-secrets/external-secrets for K8s) — deleting a hardcoded secret without a destination recreates it next sprint.

The workflow

1. 1.

   Strip values, keep shape. Preprocess the scanner JSON: drop/mask the matched secret strings, keep rule type, entropy, file path, commit age, and repo. This sanitized export is the only thing that ever enters a prompt.

2. 2.

   Classify and rank the pile.

   Copy

   Attached: sanitized secret-scan findings (no secret values) plus repo
   metadata. Classify each finding:
   - LIVE-RISK: credential type still plausibly valid (cloud key, DB
     password, API token) in an active repo
   - HISTORICAL: only in old commits, file since removed
   - TEST/FAKE: paths and rule types suggesting fixtures or examples
   - NEEDS-HUMAN: cannot determine from metadata
   Rank LIVE-RISK by blast radius (cloud admin creds > service tokens >
   webhook secrets). Group all findings by repo and owning team. Do not
   guess validity — that is checked outside this analysis.
   

   Spot-check 10 classifications before trusting the buckets — misclassified TEST/FAKE is the dangerous direction.

3. 3.

   Verify liveness out-of-band. For LIVE-RISK findings, check validity via the provider (key last-used timestamps, IAM access analyzer, DB auth logs) — never by asking the AI and never by "testing" the credential from your laptop. Update the ranking with facts.

4. 4.

   Draft per-repo remediation PRs. For each affected active repo, have the AI draft: removal of the hardcoded value, replacement with the secrets-manager reference (env var, external-secret, Vault path — per your platform's pattern), and a .gitignore/pre-commit gitleaks hook addition so the repo cannot regress. Owners review and merge; nothing auto-merges.

5. 5.

   Write rotation runbooks per credential class. One runbook per type (AWS access key, GitHub PAT, DB password...): where it is issued, every consumer that must be updated, rotation steps in order, verification step, rollback. AI drafts from your notes; the owner executes rotation by hand and ticks each step. Rotation before history-scrub — a rotated secret in history is an artifact, an unrotated one is a breach.

6. 6.

   Decide on history rewriting deliberately. For genuinely sensitive history (client repos, public repos), plan git filter-repo/BFG with the owning team — it rewrites hashes and breaks clones, so it is a coordinated change with its own record. For internal-only historical findings, rotation + documented acceptance is often the saner call.

7. 7.

   Turn the sweep into a gate. Add the scanner to CI/pre-commit org-wide so the pile never rebuilds; AI drafts the pipeline snippet and the developer-facing "you just committed a secret — do this" doc.

Tool variants

• gitleaks + chat assistant: default OSS path; gitleaks JSON is clean input for the classification prompt.
• TruffleHog: adds its own verification of liveness for many credential types — use it to reduce step 3 work, still confirm high-blast-radius finds via the provider.
• GitHub Advanced Security / GitLab secret detection: platform-native findings with push protection; use AI on the API export for the same triage and PR drafting.
• AI coding assistant in-repo: for step 4, an assistant with repo context drafts better secret-reference replacements (it can see how config is loaded) — same masked-values rule applies.

Verification

• •
  10-finding spot-check of classification passed; any TEST/FAKE misfile found means re-checking that whole bucket.
• •
  Every LIVE-RISK finding has provider-verified status recorded (valid/invalid/last-used).
• •
  Merged remediation PRs leave no plaintext secret behind (re-run the scanner on the PR branch — the scanner, not the AI, confirms cleanliness).
• •
  Each rotated credential's runbook shows all consumers updated and a post-rotation verification tick.
• •
  CI gate proven by a deliberate canary commit with a fake key — blocked as expected.

Pitfalls

• The AI touching secret values. Also covers "is this key format valid?" questions — that is pasting a secret. Preprocess first, always.
• Scrubbing history before rotating. Comforting, useless — assume anything that hit a remote is compromised until rotated.
• Trusting classification over liveness data. A years-old key can still be live; only provider records settle it.
• Remediation without a secrets-manager destination. The hardcoded value returns because the deploy needed it; fix the consumption pattern, not just the file.
• Big-bang history rewrites. Rewriting shared-repo history without team coordination breaks every clone and CI cache — it is a change-managed operation.

Restricted-environment variant

• •
  Findings metadata (repo names, file paths) from client code is itself confidential. Use a local model (Ollama) for classification — sanitized findings are small and 8B-class models handle bucket-and-rank well — or the client's approved tenant.
• •
  In client engagements, rotation of client-owned credentials usually requires a client change request: fold the AI-drafted rotation runbook into their change process, and get written sign-off on any history rewrite of client repos.
• •
  Report to the client in classes and counts, not values or paths, unless the MSA covers detail sharing.

Success criteria

• •
  The finding pile is fully bucketed with an owner and status per finding — zero "unknown" rows after a week.
• •
  All verified-live credentials rotated via runbooks, with consumer updates confirmed.
• •
  Scanner re-run on remediated repos comes back clean, and the CI gate blocks the canary commit.
• •
  Three months later the estate-wide finding count trends to near-zero new findings — the gate holds.