Wire AI Security Evals into Your CI/CD Pipeline

Tier: Core · Workflow area: AI red-team program & tooling

The discipline that separates a real AI red-team program from a one-off audit is continuous evaluation: your Promptfoo (pb-41) and Garak (pb-40) checks run automatically on every pull request that touches the prompt, the model config, the retrieval sources, or the guardrails — and a regression fails the build. This playbook wires those evals into CI/CD so security regressions are caught at the same gate as failing unit tests, by the same engineers, before merge.

When to reach for this

• •
  You have a Promptfoo suite and/or Garak probe set and want them enforced on every change instead of run ad hoc.
• •
  Your team ships prompt and guardrail changes faster than a human can manually re-red-team them.
• •
  You are standing up the "security regression harness" that AI-security job descriptions now demand.
• •
  Skip this only if the LLM surface is a throwaway prototype with no untrusted input and no path to production.

The workflow

1. 1.

   Make the suite run headless and exit non-zero on failure. Promptfoo already returns a non-zero exit code when assertions fail below threshold — that is what CI keys on.

   Copy

   # .github/workflows/ai-security-evals.yml
   name: AI Security Evals
   on:
     pull_request:
       paths:
         - "prompts/**"
         - "guardrails/**"
         - "app/llm/**"
         - "promptfooconfig.yaml"
   jobs:
     redteam:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@<pinned-sha>
         - uses: actions/setup-node@<pinned-sha>
         - run: npm i -g promptfoo
         - run: promptfoo eval -c promptfooconfig.yaml --no-progress-bar
           env:
             OPENAI_API_KEY: ${{ secrets.EVAL_MODEL_KEY }}
   

2. 2.

   Gate on path filters, not every commit. Run the full red-team suite when the prompt, guardrails, model wiring, or retrieval config change. A docs-only PR should not pay for a full scan.

3. 3.

   Set the failure threshold explicitly. Security assertions should be hard gates — 100% of the injection/leakage cases must pass. Use Promptfoo's --fail-on-error and per-assertion thresholds so a single reopened hole reds the check.

4. 4.

   Run the heavy Garak scan on a schedule, not per-PR. Full probe scans are slow; put them on a nightly or pre-release schedule and treat a drop in pass rate as a blocking release-gate rather than a per-commit gate.

   Copy

   on:
     schedule:
       - cron: "0 2 * * *"      # nightly full Garak baseline
   

5. 5.

   Publish the report as a build artifact. Upload the Promptfoo/Garak output so reviewers can open the exact failing prompt and completion from the PR, without re-running anything.

6. 6.

   Require the check to merge. Add the eval job to branch protection as a required status check. An eval that can be bypassed by clicking "merge anyway" is documentation, not a gate.

Verify before you trust it

The gate is real only if a seeded regression turns the build red. Prove it end to end: open a throwaway PR that deliberately weakens a guardrail or reverts a system-prompt hardening line, push it, and confirm the CI eval job fails and blocks merge. Then confirm a clean PR passes. This "seed a regression, watch the pipeline fail" test is the single check that certifies the whole wiring — path filters that never trigger, a non-zero exit that CI ignores, or a status check that is not actually required will all silently let a regression through, and only a deliberate failing PR exposes them. Re-run this verification whenever you change the workflow file itself.

Pitfalls

• A green check that never fails. If no seeded regression has ever reddened the job, you do not know it works. Prove it with a deliberate failing PR.
• Non-blocking status checks. An eval that is not a required check in branch protection is advisory only; regressions merge anyway.
• Flaky LLM-graded assertions in the per-PR path. Rubric variance produces random reds that train the team to ignore the gate. Keep per-PR gates deterministic; move rubric checks to scheduled runs.
• Secrets and eval-model cost in CI. Every PR run spends tokens. Use a cheap eval model, cache where possible, and scope the eval API key to the eval job only.
• Path filters that miss a surface. If retrieval sources or tool definitions can change LLM behavior but are not in the trigger paths, a regression there ships untested. Map every behavior-affecting file into the filter.

Tools

• CI/CD runner (GitHub Actions / GitLab CI)the automation host; path-filtered jobs and required status checks turn evals into a merge gate.
• Promptfoothe per-PR deterministic red-team suite that exits non-zero on regression, the natural fit for a fast gate.
• Garakthe heavier baseline scanner, run on a nightly/pre-release schedule as a release gate rather than a per-commit one.

Client-safe variant

In a client's pipeline, keep the eval provider pointed at the client's own staging endpoint and store the eval-model credential in the client's CI secret store, never yours. Where the client prohibits sending prompts to an external eval model, run the suite against a local model inside their runners and use deterministic assertions so no third-party grader is needed. Deliver the workflow file and required-check configuration as a maintainable artifact the client's platform team owns, document the seeded-regression verification so they can re-prove the gate themselves, and ensure build artifacts (which contain real prompts and outputs) are retained only within the client's environment under their retention policy.