Lab 14 · An Eval Gate in CI That Blocks a Bad Prompt

This is what turns "we tested it in the playground" into a merge gate.

Time & tier

Advanced · ~90 minutes hands-on. Capstone lab 3 of 4.

Setup

Builds on lab 13 (the /ask RAG endpoint and /triage). Work in the same capstone/.

Skipped labs 12–13? Quick-start: you need a running service with at least one LLM endpoint that (a) returns structured/parseable output and (b) has a system prompt you can edit in a file. A minimal /ask that answers from a two-doc corpus and cites [n] is enough to exercise the gate. The eval mechanics below are what's graded, not the size of the app.

Tools (OSS):

npm install -g promptfoo          # the eval runner
# service running in another terminal:  uvicorn main:app --port 8000

Also initialize git if you haven't: git init && git add -A && git commit -m "capstone pre-eval" — you need commits to see CI go red/green.

Guided steps

1. 1.

   Write the golden dataset — 30 cases. Create evals/cases.yaml covering three assertion families. Aim for ~12 format cases, ~12 groundedness cases, ~6 policy cases:

   Copy

   # each case: a request to the service + what must hold
   - vars: {endpoint: /triage, text: "my invoice is wrong"}
     assert:
       - type: is-json
       - type: javascript
         value: "JSON.parse(output).triage.category === 'billing'"
   - vars: {endpoint: /ask, question: "what is the p99 latency budget?"}
     assert:
       - type: contains        # must cite a source
         value: "["
       - type: python
         value: file://groundedness.py    # citations resolve to real chunks
   - vars: {endpoint: /ask, question: "who won the 2019 cricket world cup?"}
     assert:
       - type: contains
         value: "Not answerable from the provided sources."
   

2. 2.

   Point promptfoo at your live service. Create promptfooconfig.yaml using the HTTP provider so evals hit the real endpoint (retrieval, groundedness check, and all):

   Copy

   providers:
     - id: https
       config:
         url: "http://localhost:8000/{{endpoint}}"
         method: POST
         headers: { "content-type": "application/json" }
         # body is built from the remaining vars
         transformResponse: "json.stringify(json)"
   tests: file://evals/cases.yaml
   

3. 3.

   Write the groundedness assertion. evals/groundedness.py receives the output and returns pass/fail: parse the answer's [n] citations and the response's citations list; every cited n must resolve to a returned chunk. Reuse the check you built in lab 13 — now it's an assertion, not just runtime logic.

4. 4.

   Run the gate locally and get it green.

   Copy

   promptfoo eval -c promptfooconfig.yaml
   

   Fix cases and prompts until all 30 pass. promptfoo eval exits non-zero if any assertion fails — that non-zero exit is your CI signal.

5. 5.

   Wire it into CI. Create .github/workflows/eval.yml:

   Copy

   name: llm-eval-gate
   on: [push, pull_request]
   jobs:
     eval:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v4
         - uses: actions/setup-python@v5
           with: { python-version: "3.11" }
         - run: pip install -r requirements.txt
         - run: uvicorn main:app --port 8000 & sleep 8   # start service
         - run: npx promptfoo@latest eval -c promptfooconfig.yaml
         # non-zero exit from promptfoo fails the job
   

   (Freeze deps: pip freeze > requirements.txt. In CI you'll need the ANTHROPIC_API_KEY as a GitHub secret, or run the gate against a local/stub model to keep CI at $0 — note which you chose.)

6. 6.

   Prove the gate works — the required failure. Make a deliberately broken prompt change and commit it: in rag.py, delete the line "If the sources do not contain the answer, reply exactly: 'Not answerable...'" and the "cite the source number" instruction. Commit and push.

   • •
     The groundedness assertions fail (no citations).
   • •
     The known-absent assertions fail (the model now confabulates instead of refusing).
   • •
     CI goes red. Screenshot it.
7. 7.

   Revert and confirm green. Restore the prompt, push, watch CI pass. Red-then-green is your artifact.

Verification

• •
  promptfoo eval passes 30/30 on the good prompt and exits 0.
• •
  The broken-prompt commit produces a failing CI run (non-zero exit, job red) — this is mandatory, not optional.
• •
  Reverting produces a passing run.
• •
  The failing run's log shows which assertions failed (groundedness + policy), not a crash — the gate distinguishes a quality regression from an outage.
• •
  All three assertion families (format, groundedness, policy) are represented and at least one of each fails under the broken prompt.

Completion artifact

evals/cases.yaml, promptfooconfig.yaml, groundedness.py, .github/workflows/eval.yml, and two CI run URLs/screenshots: one red (broken prompt), one green (reverted). A short EVALS.md noting dataset composition and which assertions the broken prompt tripped. Self-grade (5 criteria = pass at 4+):

1. 1.
   30-case golden dataset spanning format, groundedness, and policy.
2. 2.
   Eval runs in CI on push/PR and fails the job on assertion failure.
3. 3.
   The deliberately-broken prompt demonstrably fails CI (red run exists).
4. 4.
   Revert demonstrably passes (green run exists).
5. 5.
   Failure output is diagnostic (names the failing assertions), not an opaque crash.

Stuck?

• promptfoo HTTP provider body/vars confusion: if templating the body is fiddly, use a python provider that just httpx.posts to your endpoint and returns the JSON — full control, no template quirks.
• CI can't reach the service: it must be started in the same job (background & + sleep) as shown; a service on your laptop isn't reachable from GitHub runners.
• Don't want to spend API credit in CI: set MODEL to a local/stub in CI, or gate on a cached-response fixture; the point is the gate mechanics, and grading doesn't require paid inference in CI.
• Broken prompt still passes: your groundedness assertion is too lenient — it must fail when citations are absent or don't resolve. Test the assertion in isolation first.

Client-safe variant

The eval harness is the client-safe crown jewel: a golden dataset + a CI gate is exactly what lets a services team promise "AI changes can't ship without passing our quality bar," which clients increasingly demand in contracts. The dataset may contain client-representative questions — keep it in the client's repo and run the gate on the client's CI (GitHub Enterprise, GitLab, Azure DevOps all run the same promptfoo eval step). If inference can't run in the client's CI, gate against recorded fixtures or the client-sanctioned in-tenancy endpoint. The transferable asset is the pattern — golden set, three assertion families, non-zero-exit-fails-the-merge — not this specific corpus.

Answer key (don't read before attempting)

The broken-prompt change (step 6) removes two things and both must be caught: (1) dropping "cite the source number" makes answers contain no [n], so every contains "[" and every groundedness assertion fails; (2) dropping the "reply exactly 'Not answerable...'" instruction makes the model confabulate on the two known-absent questions instead of refusing, so those contains "Not answerable" assertions fail. If your CI stays green after this change, the likely causes are: the groundedness assertion only checks JSON validity (not citation resolution); the policy cases use fuzzy matching that a confabulated answer accidentally satisfies; or promptfoo's exit code isn't propagating (some shells swallow it — ensure the eval is the job's final command with no || true). A subtle trap: if you lowered max_tokens or changed the model in the same commit, you'll get failures for the wrong reason — change only the prompt so the red run cleanly attributes the regression to the prompt, which is the whole point of a gate that localizes blame.