Lab 12 · Prompt Regression Gate in CI

Prompts are code: they ship behavior, they get "improved" by well-meaning teammates, and without regression tests every edit is a production experiment. Arjun's team learned this when a one-line prompt tweak — adding "be concise" — silently broke refund-eligibility answers for a whole locale. This lab rebuilds that incident in miniature: you gate a working support-bot prompt with a golden dataset in GitHub Actions, then apply a provided v2 prompt that genuinely improves three things and quietly regresses one edge case. If your gate is built right, the PR goes red with a specific failing case attached; if it goes green, the seeded regression tells you exactly which golden case you forgot to encode.

Time, tier & cost

~90 minutes hands-on · Core tier (maps to module M6 and playbook PB-37). Cost: $0 with Ollama; ≤$2 hosted. CI runs free on a public GitHub repo.

Setup

mkdir -p qa-lab-12 && cd qa-lab-12 && git init
npm install -g promptfoo
ollama pull llama3.1:8b   # or export an API key

prompts/support-v1.txt — the prompt in production today:

You are the support assistant for KiranaCart, a grocery delivery app.
Answer the customer's question using only the policy facts below.

POLICY FACTS:
- Delivery fee is ₹29; free above ₹499 cart value.
- Refunds for damaged items: photo required, must be reported within
  48 hours of delivery.
- Perishables (dairy, produce, meat) are non-returnable but ARE
  refundable if damaged, same 48-hour photo rule.
- Subscription "KiranaPlus" costs ₹99/month: free delivery on all
  orders and 5% cashback.
- We deliver 6am-11pm; no scheduled slots on public holidays.

Rules: answer in the customer's language if the question is not in
English. If the answer is not covered by the policy facts, say you
will connect them to a human agent - never guess.

Question: {{question}}

datasets/golden.yaml — the golden dataset (build the file, then extend it in step 3):

- vars:
    question: "What is the delivery fee?"
  assert:
    - type: contains
      value: "29"
    - type: llm-rubric
      value: "States delivery is free above Rs 499"
- vars:
    question: "My milk arrived spilled yesterday, can I get a refund?"
  assert:
    - type: llm-rubric
      value: "Says yes refundable if damaged, mentions the photo requirement and the 48-hour window"
    - type: not-contains
      value: "non-refundable"
- vars:
    question: "Do you deliver at midnight?"
  assert:
    - type: llm-rubric
      value: "Says no; delivery hours are 6am to 11pm"
- vars:
    question: "mera order kal kharab nikla, paise wapas milenge kya?"
  assert:
    - type: llm-rubric
      value: "Responds in Hindi (Devanagari or romanized), says refund possible for damaged items with photo within 48 hours"
- vars:
    question: "Can I return a packet of paneer I ordered by mistake?"
  assert:
    - type: llm-rubric
      value: "Says perishables are non-returnable; does NOT promise a refund since the item is not damaged"
- vars:
    question: "What's your CEO's email address?"
  assert:
    - type: llm-rubric
      value: "Does not invent an answer; offers to connect to a human agent"

promptfooconfig.yaml:

prompts:
  - file://prompts/support-v1.txt
providers:
  - id: ollama:llama3.1:8b
    config: { temperature: 0.3 }
defaultTest:
  options:
    provider: ollama:llama3.1:8b   # judge for llm-rubric asserts
tests: file://datasets/golden.yaml

Steps

1. 1Get a green local baseline. Run promptfoo eval and then promptfoo view to inspect. Fix any assert that fails for test reasons (an over-strict contains, a rubric wording the judge misreads) — but log every such fix in gate-notes.md with why it was a test bug and not a prompt bug. A golden dataset you haven't run is a guess; this step is its shakedown.
2. 2Interrogate the dataset for coverage before trusting it. Ask: which policy facts have no test? (KiranaPlus and the holiday rule are untested as provided.) Which behaviors have no test? (Language-matching has one case; the never-guess rule has one.) Note the gaps in gate-notes.md — you will feel one of them in step 5.
3. 3Extend the golden set to close the named gaps. Add at least three cases: a KiranaPlus value question ("Is KiranaPlus worth it for ₹300 weekly orders?" — rubric: mentions ₹99/month, free delivery, 5% cashback, no invented benefits), a holiday-slot question, and one adversarial phrasing of the 48-hour rule ("It's been three days but the eggs were rotten, refund?" — rubric: politely declines per the 48-hour window or offers human review; does not simply grant the refund). Rerun to green.
4. 4Wire the CI gate. Create .github/workflows/prompt-gate.yml:

name: prompt-regression-gate
on:
  pull_request:
    paths: ["prompts/**", "datasets/**", "promptfooconfig.yaml"]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm install -g promptfoo
      - run: promptfoo eval -c promptfooconfig.yaml -o results.json
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: promptfoo-results, path: results.json }

For CI, swap the provider to a hosted model via repo secret (Ollama isn't on the runner) — keep temperature and asserts identical, and note in gate-notes.md that provider-swapped CI is a pragmatic compromise you are making knowingly. Commit everything to main; confirm the workflow triggers only on prompt/dataset paths. 5. Apply the provided "improvement" on a branch. Create feat/prompt-v2, replace the prompt body's rule section with v2 below (this is the seeded change — apply it verbatim, resist pre-editing it):

Rules: be warm but concise - two sentences maximum. Always answer in
English so the whole support team can audit conversations. If the
answer is not covered by the policy facts, say you will connect them
to a human agent - never guess. End every answer with "Anything else
I can help with?"

6. 1Watch the gate do its job. Push the branch, open the PR, and read the failing eval. Expected: the Hindi golden case fails (v2 forces English), and the two-sentence cap likely breaks the damaged-milk rubric (photo + 48h + yes-refundable rarely fits warmth plus two sentences). Download the results artifact and paste the failing cases into gate-notes.md — this red run is the deliverable proof. If everything passed, your step-3 dataset didn't encode the language behavior strongly enough: strengthen the rubric (an explicit "responds in Hindi" assert), re-push, and get your red.
7. 2Fix the prompt, keep the improvement. The v2 author wanted warmth, brevity, and auditability. Negotiate all three against policy: restore language-matching (auditability via translation note, e.g. "reply in the customer's language; append a one-line English gist in parentheses"), relax the hard two-sentence cap to "as brief as possible while including all policy conditions," keep the closing line. Iterate on the branch until the full golden set is green and diff v1→v3 shows the improvements retained. Merge.
8. 3Write the gate policy. Close gate-notes.md with the team rule this lab instantiates: no prompt edit merges without a green golden-set run; every production incident adds a golden case within a day; rubric asserts get reviewed like code because they are the spec now. Include the matrix note: for high-stakes prompts, add a second provider row to the config so a model swap can't hide behind a prompt-only gate.

Verify

• •
  Baseline green run recorded; every shakedown fix logged as test-bug with reasoning.
• •
  Golden set extended to ≥9 cases, covering KiranaPlus, holidays, and the 48-hour adversarial phrasing.
• •
  CI workflow triggers on prompt/dataset paths and uploads the results artifact.
• •
  The v2 PR shows a red gate with the Hindi case (at minimum) failing, evidenced in gate-notes.md.
• •
  v3 merges green with language-matching restored and the v2 improvements retained (diff included).
• •
  Gate policy written, including the incident-to-golden-case rule.

Self-grading rubric (pass = 7/10):