Evaluation Engineering I — From Vibes to Evidence

Module 4 ended with you triaging sixty failures by hand — reading traces, attributing each to a component, building a Pareto chart with a spreadsheet and patience. That manual clinic was not busywork; it was the apprenticeship. This module industrializes it. Everything you attributed by eye — retrieval-miss, grounding-miss, generation-miss — becomes a metric computed automatically over every trace, and every failure you triaged becomes a row in a versioned golden dataset that scores your system for the rest of its life.

This is the track's signature territory: phase-1 research found evaluation engineering to be the single most-cited employer differentiator for senior DS/ML roles, with postings asking verbatim for "eval harnesses measuring accuracy, faithfulness, hallucination, and latency" and naming Ragas and Arize Phoenix as expected tools.

The Common Core evals module taught the anatomy — task set, criteria, pass bar. Here you build the production-grade version: error analysis with a formal open-coding method, datasets with provenance you can defend in a review, deterministic code-based evals as the foundation layer, and Ragas metrics you can interpret — because a faithfulness score you can't map to a pipeline component is a number, not evidence.

What You'll Learn

• The Husain/Shankar error-analysis methodopen coding on real traces, then axial coding into a failure taxonomy, with a complete worked 20-trace example
• Golden datasets with provenanceschema, versioning discipline, splits, and the refresh loop that keeps a taxonomy honest
• Code-based evals firstschema-validity rates, grounded-fact assertions, execution-based checks for SQL and numeric tasks: the deterministic layer that runs free on every commit
• The four core Ragas metricsfaithfulness, response relevancy, context precision, context recall: how each is actually computed, and a worked scoring example
• The metric → component mapwhich score pattern indicts retrieval, which indicts ranking, which indicts generation, automating Module 4's triage table

Error Analysis: The Method, Formalized

Common Core introduced the principle: look at your data before you pick metrics. At this track's altitude you need the actual method — the one Hamel Husain and Shreya Shankar turned into the de facto industry curriculum, borrowed from qualitative research. It has two phases, and the discipline is in not collapsing them.

1. 1Open coding. Pull a sample of real traces — full traces, with the retrieved chunks and any query rewrites visible, not just input/output pairs. Read each one and write a short free-text note describing what went wrong, in your own words, without trying to categorize. "Quoted the v2 schedule; runbook is on v3" is an open code. "Retrieval problem" is not — it's a premature category, and premature categories are how you miss the failure mode you didn't already believe in. Notes should be specific enough that a colleague could find the defect from the note alone.
2. 2Axial coding. Only after all notes exist do you cluster them into named failure modes, count occurrences, and rank by frequency. The taxonomy that emerges is your system's — grounded in observed behavior, not imported from a blog post's generic list.

Two operational rules make this work at team scale:

• A single benevolent dictatorone domain-informed person owns the codes and the clustering, because taxonomies built by committee converge on mush.
• Code until saturationkeep reading traces until new ones stop producing new failure modes, often somewhere between 30 and 100 traces, rather than fixing a sample size in advance.

Worked example: 20 traces from a data-platform support assistant

The system: the RAG assistant you built in M3–M4, answering data-platform questions over runbooks, wiki pages, and API docs. Twenty production traces, open-coded. (In lab L8 you'll do 100; twenty is enough to show the mechanics.)

Thirteen failures in twenty. Axial coding clusters them:

This table is the module's pivot. Reading down the rows: some failure modes are checkable by code (citation present, format, numbers match context), some are measurable by Ragas metrics (contradiction and fabrication → faithfulness; jargon misses → context recall), and one — policy compliance — will need a judge in Module 6. The taxonomy tells you which eval to build; the counts tell you in what order.

Golden Datasets: Provenance and Versioning

Every triaged failure now becomes an eval case. A golden dataset is not a pile of Q&A pairs — it is a curated, versioned artifact where every row can answer the question "why are you in this set, and who says your reference answer is right?" That property — provenance — is what separates a dataset a reviewer can trust from a JSON file someone once made.

A production-shaped row:

{
  "id": "gd-0047",
  "input": "How long are raw logs retained?",
  "reference": "Raw event logs are retained for 90 days in s3://logs-raw, then lifecycle-archived to Glacier.",
  "reference_source": "wiki/data-retention-policy#v4",
  "failure_mode": "contradicts_context",
  "origin": {"type": "production_trace", "trace_id": "tr-8821-0714", "triaged_by": "sunil", "date": "2026-07-14"},
  "expected_behavior": "answer_with_citation",
  "split": "regression",
  "status": "active",
  "notes": "System said 30 days; chunk said 90. Numeric-grounding case."
}

Design decisions that matter at this altitude:

• Origin typing. Tag every row as production_trace, expert_authored, or synthetic. Trace-derived rows carry the most evidential weight; synthetic rows fill coverage gaps (edge cases you haven't seen yet) but should be a labeled minority, because a set dominated by synthetic cases measures your imagination, not your system.
• expected_behavior includes abstention. Trace 19 taught the lesson: your set needs unanswerable questions whose correct output is "I don't know / not in the docs." A system evaluated only on answerable questions is being trained by your eval to never abstain. These rows become your hallucination-rate slice in Module 6.
• Splits with a purpose. A regression split (known failures — the system must keep passing these), a holdout split you never tune against and consult only for adoption decisions, and a smoke subset small enough to run in seconds during iteration.
• Append-and-deprecate, never edit-in-place. When a reference answer turns out wrong, set status: deprecated and add a corrected row. Editing rows silently rewrites history and breaks score comparability across versions.
• Version like code, because it is. The dataset lives in git as plain JSONL/CSV with a changelog; tag releases (golden-v1.3) and record the dataset version alongside every eval score. For large sets or binary attachments, DVC handles the storage; MLflow and Phoenix both support registered dataset versions tied to eval runs — use whichever your stack already traces to, but git remains the source of truth for the rows.
• Refresh on a cadence. Error analysis is not a launch event. A standing rhythm — say, triage 20 fresh production traces weekly — feeds new failure modes into new rows. Datasets are grown, not designed; a golden set that hasn't changed in a quarter is measuring last quarter's system.

Code-Based Evals First

Before any LLM grades anything, extract every check that code can decide. Deterministic evals are free, instant, perfectly repeatable, and immune to judge bias — which is why the practitioner hierarchy (Husain, Yan, and every mature eval stack) puts assertions above judges wherever the task has checkable structure. Your taxonomy above already shows the seams: five of the eight failure modes are partly or fully code-checkable.

Schema-validity rate. For structured outputs, the fraction of responses that parse and validate against the Pydantic schema — the reliability metric you built in M2's extraction service, now formalized as an eval that runs per-change rather than per-incident.

Grounded-fact assertions. Numbers, dates, and identifiers in the answer must appear in the retrieved context — a direct, deterministic counter to traces 14 and 17's failure class:

NUM_PAT = re.compile(r"\d[\d,.]*")

def eval_numbers_grounded(case, trace) -> bool:
    nums_out = set(NUM_PAT.findall(trace.answer))
    nums_ctx = set(NUM_PAT.findall(" ".join(c.text for c in trace.retrieved)))
    return nums_out <= nums_ctx          # every number in the answer exists in context

def eval_citation_present(case, trace) -> bool:
    return bool(re.search(r"\[source:.+?\]", trace.answer))

Execution-based checks. For generated SQL: does it parse (sqlglot), does EXPLAIN succeed against the read-only replica, does executing it return the reference row count or reconcile to the reference aggregate within tolerance? For generated analysis code: does it run in the sandbox and reproduce the pinned number? Execution is the strongest ground truth available — a query that runs and reconciles needs no judge.

Behavioral assertions. Response length bounds, required-format compliance (trace 10), refusal-marker present on the unanswerable slice, latency and token cost per case — cost and latency are eval metrics too, and employers ask for them by name.

Wire these as a pytest-style harness over your golden set: each case is parametrized, each assertion is a test, and the output is a per-case, per-check matrix rather than one blended score. This harness is the CI backbone Module 6 will gate merges with — judges get added on top of it, never instead of it.

Ragas Metrics: What Each Score Actually Means

Code checks can't grade the semantic core of RAG quality — whether the answer is supported, relevant, and built from good retrieval. That's the Ragas layer. Ragas metrics are themselves LLM-computed (each metric makes judge-model calls under the hood — budget accordingly and use a cheap-tier model), but they are structured judgments with defined decompositions, which makes them interpretable in a way a raw "rate 1–10" judge is not. Four metrics carry most of the weight; know their mechanics, because the mechanics are what make the scores mean something.

• Faithfulnessis the answer supported by the retrieved context? Mechanics: decompose the answer into atomic claims, then check each claim for support against the retrieved chunks; the score is supported claims ÷ total claims. A faithfulness of 0.60 means, concretely, that two of every five claims in the answer have no basis in what was retrieved. This is the hallucination-adjacent metric — trace 7 and trace 14 fail it hard.
• Response relevancy (answer relevancy) — does the answer address the question asked? Mechanics: generate several plausible questions from the answer, embed them, and average their cosine similarity to the original question. An evasive, incomplete, or off-target answer generates reverse-questions that don't match the user's. Trace 5 — half the question dropped — scores low here while potentially scoring perfectly on faithfulness, which is exactly why you need both: faithful-but-irrelevant and relevant-but-unfaithful are different diseases with different cures.
• Context precisionof what you retrieved, how much was worth retrieving, and was it ranked well? Mechanics: judge each retrieved chunk as relevant/irrelevant to answering, then average precision@k over the relevant positions — so relevant chunks ranked high score better than the same chunks ranked low. Worked: three chunks, relevance flags [1, 0, 1] → precision@1 = 1.0, precision@3 = 2/3; score = (1.0 + 0.667) / 2 ≈ 0.83. Drop the gold chunk to rank 3 ([0, 1, 1] with one relevant distractor) and the score falls — the metric sees ordering, not just membership.
• Context recalldid retrieval surface everything the reference answer needs? Mechanics: decompose the reference answer into claims and check what fraction is attributable to the retrieved context. This is the only one of the four that requires a reference answer — which your golden dataset's provenance discipline just paid for. Context recall is the automated version of M4's "is the gold chunk in the top-k?" interrogation step.

Worked scoring pass

Query: "How long are raw logs retained, and where do they live?" Retrieved: [retention-policy#c2 (gold), faq-billing#c7 (noise), s3-layout#c4 (gold)]. Answer: "Raw logs are retained for 90 days. They are stored in s3://logs-raw. After retention they are deleted immediately. Retention is configured by the platform team."

• •
  Claims: (1) 90-day retention — supported by retention-policy#c2. (2) stored in s3://logs-raw — supported by s3-layout#c4. (3) deleted immediately after retention — context says lifecycle-archived to Glacier: unsupported. (4) configured by platform team — supported. Faithfulness = 3/4 = 0.75.
• •
  Reverse-questions from the answer match the original well → response relevancy high (~0.9).
• •
  Chunk relevance [1, 0, 1] → context precision ≈ 0.83 as computed above.
• •
  Reference claims (90 days; s3://logs-raw; Glacier archival) — Glacier detail is present in retention-policy#c2, so all 3 attributable → context recall = 1.0.

Diagnosis reads itself: retrieval did its job (recall 1.0), ranking let one distractor in (precision 0.83), and the generator invented a deletion claim despite the evidence being present — a grounding-miss. Fix the generation prompt, not the retriever.

The metric → component map

This table is Module 4's triage procedure, automated — commit it to memory, because the track's assessment questions live here:

Thresholds are yours to set from your own baseline, but a widely used starting heuristic: faithfulness below ~0.70 on any slice is an investigate-now signal for hallucination, and any metric's drop against your own last release matters more than its absolute value. Score every metric per failure-mode slice, never only in aggregate — "faithfulness 0.91 overall, 0.58 on multi-part questions" is a roadmap; "0.87 overall" is a shrug.

Check yourself: Your dashboard shows faithfulness at 0.88 overall and you're tempted to call the system healthy. Why is that number a shrug rather than evidence, and what would make it actionable?

Exercises

1. 1Open-code 20 of your own traces. From your M3–M4 pipeline (or the lab's provided trace pack), open-code 20 traces exactly as in the worked example — free-text notes first, no categories. Then axial-code into a taxonomy with counts and map each failure mode to its M4 component. Deliverable: both tables, plus one sentence on any failure mode the M4 taxonomy has no bucket for.
2. 2Mint a 15-row golden dataset with provenance. Convert your taxonomy into golden-set rows using the JSON schema above: at least 10 trace-derived, at most 5 synthetic (labeled), at least 2 unanswerable-abstention cases, splits assigned. Commit to git with a changelog entry, tag it golden-v0.1.
3. 3Build the deterministic layer. Write a pytest harness with at least four code-based checks over your golden set (number-grounding, citation-format, schema validity, and one execution- or refusal-based check). Report the per-case matrix and the pass rate per check.
4. 4Score and attribute. Run faithfulness, response relevancy, context precision, and context recall over your golden set with a cheap judge model. For your three worst-scoring cases, write the component attribution using the metric → component map and state the single fix you'd try first — then check whether it matches what your manual open-coding said about the same traces.

Your playbooks & labs

This module's playbook library packages the reusable assets: the open-coding worksheet and axial-coding procedure card, the golden-dataset row schema with provenance fields and the append-and-deprecate versioning policy, the pytest harness template with the four deterministic check families, and the Ragas metric-interpretation card with the metric → component map and slice-reporting template. Do this at work this week → pb-34 Error Analysis First: 100 Traces to a Failure Taxonomy and Golden Dataset (Neha's Monday ritual, formalized), pb-35 RAG Evaluation with Ragas and Claim-Level Hallucination Checks (the component metrics you can interpret), and pb-37 Code-Based Evals and CI Regression Gates for LLM Changes (the deterministic layer that runs free on every commit). The anchored lab is L8 — error analysis → golden dataset: you read 100 real traces of a support bot, open-code them to saturation, and ship a versioned 50-item golden set with provenance plus a runnable eval script; grading weighs taxonomy quality and provenance discipline as heavily as the code. L8's output is load-bearing — Module 6 calibrates its judge against these labels, and its CI gate runs this exact dataset, so cutting corners here is borrowing pain from next module.

Further Reading

• •
  Hamel Husain — Your AI Product Needs Evals — the error-analysis-first methodology this module's workflow is built on
• •
  Ragas — Metrics concepts — official definitions and mechanics for faithfulness, response relevancy, context precision, and context recall
• •
  Ragas — Evaluating RAG pipelines — official end-to-end evaluation walkthrough
• •
  Arize Phoenix documentation — OSS tracing plus the trace→dataset workflow used to mint golden rows from production
• •
  MLflow — GenAI evaluation and datasets — dataset versioning tied to eval runs, if MLflow is your tracking stack
• •
  DVC documentation — data versioning for golden sets that outgrow plain git
• •
  DeepEval documentation — pytest-style eval framework that maps directly onto this module's harness pattern