The weekly quality report has two parts: numbers, which must be exact, and narrative, which must be written. LLMs are terrible at the first and excellent at the second — so this workflow pins every number to a queryable source, lets AI write only the surrounding prose, and mechanically catches any digit the model touched.
Tier: Foundation · Workflow area: Quality leadership & governance
This playbook builds a report pipeline: metrics pulled from TestRail/Xray and CI APIs into a frozen JSON, prose generated under a strict no-new-numbers contract, and a mechanical diff that proves every figure in the narrative traces to the source file. The result reads like your best test lead wrote it — because the judgment calls still are theirs.
A weekly or per-release quality summary consumes hours of a lead's week. Stakeholders complain reports are stale, inconsistent between authors, or arrive after the decisions they should have informed. Trend explanations ("why did pass rate dip?") are wanted but nobody has time to write them.
Your test-management data is unreliable (stale cases, unrecorded runs) — automating narrative over bad numbers launders garbage into confident prose. A dashboard already answers the question and nobody reads the prose — kill the report instead of automating it.
Decide the metrics contract first. With the report's consumers, fix the exact figures the report carries — pass rate, run counts, new/closed defects by severity, escaped defects, flake rate, coverage of planned scope, cost of the AI-eval suite if you have one — and their precise definitions (does "pass rate" include skipped? blocked?). Ambiguity here becomes confident wrong prose later.
Extract to a frozen, source-tagged JSON. A script queries each API and writes one file per reporting period; every value carries its provenance:
{
"period": "2026-W32",
"metrics": {
"pass_rate": {"value": 94.2, "unit": "%",
"source": "testrail:run/2214", "prev": 97.1},
"new_defects_sev1": {"value": 2,
"source": "jira:JQL-weekly-sev1", "prev": 0},
"flake_rate": {"value": 3.1, "unit": "%",
"source": "ci:flaky-job-report#88", "prev": 3.4}
},
"notes_from_lead": [
"Sev1 x2 are both in the new payments flow, fixes in review",
"Pass-rate dip traced to env outage Tue, not product regressions"
]
}
The notes_from_lead array is where human judgment enters — two minutes of bullets from the person who knows why the numbers moved. This is the highest-value input in the pipeline; the model can describe a dip but only the lead knows it was Tuesday's environment outage.
Generate prose under the no-new-numbers contract.
Write the weekly quality report from the attached JSON.
Hard rules:
- Use ONLY numbers present in the JSON, verbatim. Do not compute
deltas, percentages, averages, or totals — if a delta matters
and is not in the JSON, write "[delta not computed]".
- Every number you mention must be immediately followed by its
source tag in parentheses, e.g. (testrail:run/2214).
- Explanations for changes may come ONLY from notes_from_lead.
If a change has no note, say "cause not yet analyzed" — do not
hypothesize.
- Structure: 3-sentence executive summary; Trends; Risks &
actions; one-line outlook. Max 350 words.
- Match the tone of the attached example report.
The "do not compute" rule looks extreme and is the whole trick: LLM arithmetic on report numbers is where credibility dies. If you want deltas, compute them in the extraction script and put them in the JSON — which is why each metric carries prev.
Verify mechanically before any human reads it. A 20-line script extracts every numeric token from the generated prose and checks it exists in the JSON:
python verify_report.py report_2026-W32.md metrics_2026-W32.json
Numbers found in prose: 11
Matched to source JSON: 10
UNMATCHED: "5.2%" (line 14) — not present in metrics file
VERDICT: FAIL — regenerate or hand-fix line 14
An unmatched number means the model computed or invented one despite instructions — it happens, which is exactly why the verifier exists and is not optional. Green verifier, then human review.
Add the anomaly-flag pass. A second, separate prompt reviews the JSON alone: "List any metric whose change vs prev looks anomalous or whose combination is suspicious (e.g., pass rate up while run count halved). Flag; do not explain." Its output goes to the lead as questions before the report ships — the model is good at "this pattern is odd," and the human is the one who may say why in notes_from_lead.
Ship on schedule, keep the archive. Wire extraction + generation + verification into a scheduled CI job that opens a PR with the draft; the lead reviews, edits, merges, and the merge publishes. The archive of JSON files becomes its own asset: quarter-end reviews regenerate summaries across 13 weeks of frozen metrics with the same no-new-numbers contract.
Put the source tags in the published report, not just the draft. Stakeholders click through exactly once, discover the numbers are real, and the report inherits a credibility that hand-written summaries spend years earning.
prev values ship so no delta is ever model-math.Pin the numbers, generate the prose, verify the join. The lead's judgment enters as two minutes of bullets instead of two hours of writing — which is the correct division of labor between a human who knows why and a model that writes well.