Evaluation, Fine-Tuning Data & Governance

Every module since M5 has asked you to make a judgment call — chunking strategy, embedding model, hybrid weights, extraction prompts — and this module is where judgment calls become measured decisions. You'll build the RAGAS evaluation harness that gates every pipeline change in CI, run the data factory behind fine-tuning, classify PII at warehouse scale, put AI to work on the catalog itself, and wrap the whole platform in the compliance patterns that client work demands.

The scenario — the Friday deploy the eval harness caught

Meena runs the RAG data platform at a services company's GCC in Hyderabad, serving an insurance client's policy-document assistant.

That is the same report format Meena's client now receives monthly, because the MSA's AI annexure asks for documented retrieval-quality metrics, and because India job postings for exactly Meena's role now list "documented RAGAS scores" as a shortlisting differentiator. Nothing about the platform got smarter over the weekend. What changed is that a pipeline decision — the same class of decision M5–M8 taught you to make — went from an opinion with a cost spreadsheet to an experiment with a number, and the bad version of it cost one blocked merge instead of one client escalation.

That harness, and everything that has to be true around it — the golden set, the data factory feeding fine-tuning, the PII tags, the catalog, the compliance file — is this module.

RAGAS — a number on every pipeline decision

The golden set comes first, because every metric is computed against it. A golden retrieval set is a list of questions paired with the chunks (or documents) that should be retrieved — 50 questions is enough to start, 100–150 stratified across document types and query styles is a mature set. Source them from real user queries where you can, synthesize the rest (next section), and hand-verify every ground-truth label: an unverified golden set just automates your blind spots.

With the set in place, RAGAS scores the two halves of the system separately — and the split is the point, because it tells you which layer to fix:

Context precision and recall are the DE-owned half, and they're the reason this harness belongs in your CI, not just the app team's. The discipline that makes it compound: run the same harness over every variant of every choice. The M5 chunking bake-off, the M6 hybrid-vs-dense comparison, the M8 semantic-layer curation, Friday's embedding swap — one harness, one golden set, one comparable number per variant. Chunking choice alone can swing recall ~9% on the same corpus; without the harness you'd attribute that swing to the model.

Then make it a gate, not a report:

# tests/test_retrieval_regression.py — runs on every PR touching the pipeline
def test_retrieval_quality_gate():
    result = evaluate(
        dataset=golden_set,               # 124 question -> ground-truth-context pairs
        metrics=[context_precision, context_recall],
    )
    assert result["context_precision"] >= 0.75, result
    assert result["context_recall"]    >= 0.80, result

Thresholds come from your current baseline minus an agreed tolerance — the gate exists to catch regressions, not to enforce a vanity number. Track every run in Arize Phoenix or Langfuse so the metric history lives next to the traces, and remember the division of labor from M7: canaries catch production staleness continuously and cheaply; this harness measures quality and gates changes in CI. Teams running only one of the two get burned by the other's failure mode.

The golden set is a maintained asset, not a one-time artifact — version it in git next to the pipeline code, tag each item with source (user_query vs synthetic) and document type, and review it on a cadence tied to corpus change: new document types get new questions before their first index build, retired documents take their questions with them, and questions that every variant answers perfectly get replaced with harder ones, because a saturated benchmark measures nothing. When production query logs show a cluster of questions the golden set doesn't represent, that's the refresh signal — the set should drift with real usage, deliberately and under review, never silently.

The fine-tuning data factory

When an org fine-tunes a model, the training run is a commodity; the dataset is the product, and building it is data engineering. Netflix-scale post-training runs on exactly these pipelines. The factory is a funnel from raw material (support logs, documentation, annotated Q&A) to validated training JSONL, and every stage records its counts:

1. 1Exact deduphash-based, cheap, first. Duplicate examples over-weight their pattern in training.
2. 2Near-dup removal via embeddingsembed every example, ANN-search for neighbors above ~0.9 cosine, keep one per cluster. This is the M5/M6 toolkit pointed at training data.
3. 3Quality filteringheuristics first (length bounds, language detection, truncation artifacts), then LLM scoring for the judgment layer:

class QualityScore(BaseModel):
    correctness: int        # 1-5: is the answer actually right?
    completeness: int       # 1-5
    tone_fit: int           # 1-5: matches the target assistant's style
    keep: bool
    reason: str

response = client.messages.parse(
    model="claude-opus-4-8",
    max_tokens=1024,
    thinking={"type": "adaptive"},
    messages=[{"role": "user", "content": render_example(qa_pair)}],
    output_format=QualityScore,
)
score = response.parsed_output

Calibrate the judge against ~100 human-scored examples before trusting it at scale, and run the corpus through the batch API — quality-scoring 80K examples is a batch job, not a loop. Argilla is the vendor-neutral pick when the review band needs a human UI.

4. 1PII scrubbingbefore anything else leaves your boundary, and non-negotiable for client data (next sections). A name that survives into a training set is effectively permanent.
5. 2Format validationprovider format validators on every JSONL line; one malformed record can fail a run you queued for hours.
6. 3Leakage-checked splitsthe step most teams skip and most teams regret. Splitting after dedup isn't enough: run the near-dup check across the train/eval boundary, because a paraphrase of a training example sitting in the eval set inflates your eval into fiction.

The deliverable is the dataset plus a data datasheet: counts at every funnel stage (raw 120K → exact-dedup 96K → near-dup 81K → quality-pass 62K → PII-clean 62K → final splits), filter thresholds, judge calibration numbers, and sample audits. That one page is what makes the dataset defensible to a client, an auditor, or the next engineer.

Synthetic data — generation is easy, gating is the job

Synthetic data feeds both factories: golden questions for the RAGAS harness and instruction pairs for fine-tuning. Generation is one prompt — RAGAS's testset generator will manufacture question→ground-truth pairs straight from your corpus; distilabel does the instruction-pair equivalent. Ungated synthetic data, though, converges on the generator's favorite phrasings and easy questions, and quietly poisons whatever consumes it.

So the pipeline is generate → gate, with three gates minimum:

• LLM-as-judge filterthe same QualityScore pattern, tuned for "is this question answerable from this chunk, non-trivial, and unambiguous."
• Near-dup removalacross the synthetic pool and against real data.
• Distribution checksquestion-type mix, length distribution, per-document coverage, against targets you set, so 400 easy factoid questions don't masquerade as a balanced eval set.

Keep synthetic and human-labeled items tagged separately in the golden set; when a synthetic question behaves oddly in the harness, you want to know its provenance in one query.

LinkedIn's fastest-growing-roles list has Data Annotator at #4 — which is the market telling you what this skill is worth: engineers who automate annotation with quality gates are leveraged; pipelines that skip the gates just generate slop faster.

PII classification at warehouse scale

Every governance obligation downstream — masking, ACLs, DPDP compliance, the scrub stage in the factory above — depends on knowing where personal data lives, and at 1,900 tables nobody knows by hand. Two passes, then a human:

1. 1Warehouse-native classifiers first. Snowflake's sensitive-data classification and BigQuery + Cloud DLP scan column names and sampled values in place — data never moves, results land as tag proposals. This is the cheap, governed baseline.
2. 2LLM batch pass for what they miss. Native classifiers are pattern-driven and miss context: a notes free-text column full of phone numbers, a ref_code that's actually Aadhaar-derived, PII living inside JSON blobs. Send column name + type + 20 sampled values (sampled in situ — warehouse-native AI functions shine here, because sampling PII out to an external API to ask whether it's PII is its own compliance incident) and force a structured verdict: {pii_type, sensitivity, confidence, evidence} over your tag taxonomy.
3. 3Human approval, always. Proposals flow into the catalog as suggested tags with confidence attached; a data steward approves or rejects, and rejections feed back into the prompt's few-shot examples. Auto-applied tags fail in both directions — missed PII leaks, over-tagged columns grind analytics to a halt — and approval is what makes the tags authoritative enough to drive policy.

Approved tags must then do something: masking policies and row-access rules keyed to tags, exclusion from text-to-SQL retrieval and results (the M8 blocked-tables guardrail), and ACL propagation into RAG indexes (M7's allowed_groups metadata). Re-run classification on schema change — new columns arrive untagged, and untagged must mean restricted-by-default, never invisible.

AI for the catalog itself — enrichment and reconciliation agents

The same agentic patterns you've been applying to pipelines apply to metadata, with the same human-in-the-loop shape as PII tagging.

The catalog enrichment agent walks the catalog for gaps — missing descriptions, absent owners, stale tags — and drafts proposals from evidence it can already reach: lineage (what feeds this table, what consumes it), query logs (who queries it, joined to what, filtered how), dbt model SQL and docs. Drafts land in an approval queue; stewards approve, edit, or reject; coverage is measured in CI like test coverage. Atlan's AI features and Unity Catalog + Genie are the packaged versions; a custom agent over your catalog's API via MCP is the build-your-own route. The approval queue is load-bearing — an auto-written catalog converges on plausible-sounding descriptions nobody verified, which is worse than an empty field because it looks authoritative.

The reconciliation agent (the ops sibling) runs on schedule — Airflow's Common AI provider gives you LLM/agent operators natively, or a custom Claude + MCP agent — comparing source-system aggregates against warehouse marts: row counts, sums by day, key metrics. On drift, it doesn't just alert; it drafts the investigation — ranked hypotheses from lineage plus recent-change history, the confirming queries for each, and a ticket with the evidence attached. A concrete run: the nightly job finds orders in the source at 1,284,411 rows for yesterday but marts.orders_daily reporting 1,271,062; the agent walks lineage, notices a schema change merged at 15:20 on the staging model, drafts "13,349-row gap correlates with PR #198 adding a status filter — confirming query attached," and files the ticket. The M3 discipline applies unchanged: the agent proposes and evidences; a human confirms before anything is "the root cause."

Client data and compliance — the patterns that keep the contract

For the services engineer, this section is the difference between an AI capability you can sell and one that breaches an MSA. Most client contracts prohibit pasting client schemas or data into public AI tools — so the first artifact is the approved-channel hierarchy, written down and enforced, not folklore:

Layer the regulatory map on top. India's DPDP Act 2023 (with the 2025 draft rules) attaches obligations to personal-data processing that your pipelines perform, and its cross-border provisions are triggered exactly where this track lives: a US-hosted LLM API processing Indian personal data. Client-side regimes stack on — GDPR for EU data, HIPAA for health, RBI/SEBI data-localization for BFSI, the largest Indian services vertical. The engineering consequences are patterns you now already own:

• PII scrubbing before any external LLM stageM8's batch extraction over customer documents is precisely where this bites.
• Warehouse-native processing as the compliance-friendly defaultCortex AISQL, BigQuery AI functions, and Databricks ai_query keep processing inside the governed tenancy, a genuine differentiator when pitching regulated Indian clients.
• Audit logging on every AI touchpointextraction runs, in-SQL function calls, text-to-SQL queries, MCP calls; because "show us every AI system that touched personal data" is a question DPDP lets a regulator ask.
• Erasure that reaches everythingM7's delete-propagation discipline extended to training sets, eval fixtures, and synthetic pools derived from the deleted data.

The capstone — one platform, evaluated and governed

The capstone assembles M5–M8 into a single system and then proves it with this module's machinery. Concretely: the M5 ingestion/chunking/embedding pipeline and M6 hybrid retrieval with reranking, operated under M7's sync, freshness monitoring, and ACL filtering, feeding M8's serving layer — extraction tables behind validation gates and text-to-SQL behind its guardrail matrix — the whole thing wrapped in an M9 governance layer: a hand-verified golden set with RAGAS precision/recall tracked per pipeline decision and gated in CI, PII tags approved and enforced down to the indexes, and a compliance file (channel hierarchy, audit-log samples, erasure runbook) that would survive a client review. The deliverable is a repo plus an evidence pack: the RAGAS report per major decision, the guardrail matrix, the data datasheet, the deletion certificate. Any candidate can demo a RAG bot; the portfolio that gets shortlisted — and the phrase India postings literally use is "documented RAGAS scores" — is the one where every architectural choice has a number attached and every governance claim has an artifact behind it.

Check yourself: A pipeline PR drops RAGAS context precision by 18 points while faithfulness is unchanged. Which layer is at fault, who owns the fix, and why does the metric split matter?

Where this lands in your week

This module anchors LAB-15 — the fine-tuning data factory: take a deliberately messy Q&A corpus through the full funnel — embedding-based near-dup removal, calibrated LLM quality scoring via the batch API, PII scrubbing, JSONL format validation, and leakage-checked splits — then generate a RAGAS synthetic eval set from the same corpus behind judge, dedup, and distribution gates. Artifact: the curated dataset plus the data-quality datasheet with counts at every filter stage and sample audits. The capstone then folds in your artifacts from LAB-07 through LAB-14, adds the CI regression gate, the approved PII tag set, and the compliance file, and produces the evaluated, governed RAG data platform that is this track's flagship portfolio piece. Everything runs on the free path — RAGAS, Phoenix, and Langfuse are OSS; the warehouse work fits BigQuery sandbox or Databricks Free Edition; batch-API spend stays inside the ~$5-per-lab budget — with the one paid assistant subscription remaining the track's only recurring cost. Do this at work this week → pb-52 Evaluate RAG Retrieval Quality with a RAGAS Golden Set, pb-50 Build a Fine-Tuning Dataset Curation Pipeline, and pb-51 Generate Synthetic Training and Eval Data with Quality Gates.