This module installs the review reflex for AI-generated analysis code — splits first, preprocessing second, feature timing third, metrics last — so a script that runs clean on a beautiful number gets falsified in six minutes instead of shipped.
The churn model's notebook lands in Neha's review queue, and the metrics are beautiful: 0.91 accuracy, an AUC that would make the quarterly review sing.
Karthik generated the whole train-and-evaluate script with an assistant — feature engineering, scaling, an XGBoost baseline — and it runs clean end to end.
Priya, the staff data scientist who never met a number she wouldn't interrogate: "0.91 on churn? We've never cracked 0.7. What's different?" Neha doesn't answer yet.
She reads the script in a fixed order she drilled into herself years ago. The data was sorted by obs_date, then shuffled into a random 80/20 split: future rows training a model graded on the past, the same customers on both sides.
The scaler was fit before the split.
A refund_count feature over an unspecified window.
Accuracy, on a 9-to-1 imbalanced target. She has her answer for Priya: the 0.91 is fiction, and she can name all four reasons why.
Twenty minutes to generate; six minutes to falsify.
That gap — between code that runs and analysis that is true — is the discipline this module installs. You already prompt well, you know the trust tiers, and you know that "the model wrote it" carries no professional weight — that was the Common Core. This module applies that discipline to the specific work of a data scientist with years of pandas, SQL, and scikit-learn muscle memory.
The market context makes the stakes concrete: the BLS projects Data Scientist employment growing 34% through 2034, LinkedIn's Jobs on the Rise 2026 puts AI Engineer at #1 with postings up 143% year over year, and employer requirements have shifted hard — postings mentioning LLM and RAG skills are up over 340% since 2024, and Indian AI/ML hiring rose 25% year over year on Naukri's JobSpeak index.
Yet the daily reality is captured by a different number: 66% of developers in the 2025 Stack Overflow survey report spending more time than expected fixing AI code that is "almost right, but not quite." For analysis code, "almost right" is worse than wrong — a syntax error stops you; a silently leaked feature ships a model whose offline metrics are fiction. This module builds the working system: where AI genuinely delivers 2–3x on EDA, cleaning, SQL, and statistical work; the defect taxonomy that catches what generated analysis code gets wrong; and the agentic workflow that turns a notebook into a production package without a week of hand-refactoring.
The differentiating skill is not generating analysis code faster — it is catching the defects that AI-generated analysis code characteristically contains: leakage, bad splits, wrong tests, silent coercions. Speed without that catch rate is negative productivity with a delay.
The high-win zone for a data scientist has a consistent shape: bounded scope, output you can verify against the data itself, and low blast radius. Route deliberately.
The first hour with a new dataset — dtypes, null patterns, cardinality, suspicious distributions, duplicated keys — is mechanical and fully verifiable, which makes it ideal delegation material. The pattern that works is generate the profiling code, not the profile: have the assistant write the code that computes the answers, run it yourself, and read the output against the data. A prompt shape that produces useful profiling code instead of generic boilerplate:
Write a profiling script for a pandas DataFrame loaded from
transactions.parquet. I care about, in order:
1. Keys: is (customer_id, txn_ts) unique? Show violation examples.
2. Nulls: per-column null rate, AND null co-occurrence (which columns
go null together — that pattern usually means a failed join).
3. Ranges: numeric columns with negative values, zeros where zeros are
suspicious (amount, quantity), timestamps outside 2019-2026.
4. Categoricals: cardinality, top-10 values, values that look like
encoding failures (mojibake, 'nan' as string, mixed case dupes).
Output a findings dict I can assert against, not prints.
Note the last line: asking for assertable output instead of prints converts the profile into the seed of a data-validation suite (you will formalize this with pandera later in this module). Combine with an open profiler — ydata-profiling for the broad sweep, generated code for the dataset-specific questions the profiler cannot know to ask.
The cleaning step that follows is where the first defects appear. AI-generated cleaning code loves dropna() and fillna(0) — both are modeling decisions wearing a cleaning costume. Your standing rule: every imputation, drop, or coercion the assistant proposes gets a one-line justification in the code comment, written by you, or it does not ship.
The 2–3x on EDA does not come from the model writing df.groupby(...) faster than you type it. It comes from parallelizing hypothesis generation: state what you know, ask for competing explanations, and generate the analysis code for each in one pass.
Context: weekly activations dropped 18% in the last 4 weeks for the
IN region only; other regions flat. Tables: activations (daily grain),
campaigns, app_releases, pricing_changes.
Propose the 6 most likely explanations, ranked. For each: the exact
pandas analysis that would confirm or kill it, and what result pattern
distinguishes it from the others. Do not write conclusions — write the
discriminating analyses.
You then run the analyses — all of them, including the ones for hypotheses you privately favor against. This is UC-02's discipline: the model proposes, the data disposes, and you never accept a generated conclusion that you have not seen the numbers for. When you draft the findings memo, pin the numbers: paste the actual computed values into the prompt and instruct the model to use only those figures. Summarization with unpinned numbers is how a 17.8% drop becomes "roughly 20%" becomes "a fifth of activations" in the exec deck.
Never let an assistant both compute a number and report it in prose without you seeing the intermediate value. Rounding drift and outright hallucinated statistics survive review precisely because the surrounding prose is fluent.
Text-to-SQL for your own analytical questions is now boilerplate — warehouse copilots in Databricks, BigQuery, and Snowflake ship it natively. The expert friction point is that generated SQL fails quietly: it runs, returns plausible rows, and is wrong. Three standing verifications, in order of catch rate:
SELECT COUNT(*), a known monthly revenue figure, a row count from the source system — and check the generated query's result is consistent with it. Fan-out from a bad join key is the single most common generated-SQL defect, and reconciliation catches it in seconds.EXPLAIN on anything touching large tables. Generated queries favor correctness-shaped patterns (correlated subqueries, DISTINCT as duplicate-hiding) that are 100x cost on warehouse pricing.Feed the schema, not the database: paste CREATE TABLE statements plus three sample rows per table into context. Schema-in-context prompting is the difference between SQL that references your actual column names and SQL that invents customer_lifetime_value because it usually exists.
For test selection and experiment design, the model's value is adversarial, not authoritative. You have 4–15 years of statistical judgment; use the model to attack it:
I plan to analyze this A/B test with a two-proportion z-test on
conversion, n=48k per arm, randomized at user level, 3-week run.
Argue against this design. What assumptions am I making that could
be false here? What would a skeptical reviewer check first —
SRM, interference, ratio-metric issues, peeking? Be specific
about which diagnostics to run on THIS design.
The "argue against this" framing matters: asked to validate, models validate; asked to attack, they surface the sample-ratio-mismatch check you skipped. The failure mode to guard is subtler — the model will confidently recommend a reasonable-sounding but wrong test (a t-test on a heavily zero-inflated revenue metric, a chi-square where expected cells are tiny). Assumption-checking code is delegable; the final test selection is not. That call stays yours, and you should be able to defend it without citing the model.
Interview processes have already priced this in — AI-enabled formats at major employers score whether you catch planted defects in generated code, and roughly half of applied-DS interview loops in this program's research probe exactly this. More importantly, production incidents come from a short, learnable list. Generated analysis code fails in characteristic ways because the model reproduces the most statistically common patterns in its training data — and the most common patterns in public notebook code are subtly wrong.
scaler.fit_transform(X) ten million times; it has seen the pipeline-correct version far less.errors='coerce' scattering NaNs, integer division truncating, timezone-naive comparisons against timezone-aware columns, merges silently switching to cartesian behavior.This is the drill format from UC-15, and the exemplar of the review discipline. It is exactly what Neha did to Karthik's script in the cold open — here is the code the assistant produced for "build a churn model baseline with proper evaluation":
df = pd.read_csv("churn.csv", parse_dates=["signup_date", "obs_date"])
df = df.sort_values("obs_date")
# Feature engineering
df["days_active"] = (df["obs_date"] - df["signup_date"]).dt.days
df["support_contact_rate"] = df["support_tickets"] / df["days_active"]
# Preprocessing
scaler = StandardScaler()
features = ["days_active", "support_contact_rate", "monthly_spend",
"last_30d_logins", "refund_count"]
df[features] = scaler.fit_transform(df[features]) # <- Defect 1
X_train, X_test, y_train, y_test = train_test_split(
df[features], df["churned_next_90d"],
test_size=0.2, random_state=42) # <- Defect 2
model = XGBClassifier(n_estimators=300).fit(X_train, y_train)
print("Accuracy:", model.score(X_test, y_test)) # <- Defect 4
The review, as you should narrate it:
Pipeline([("scale", StandardScaler()), ("clf", XGBClassifier())]) so fitting happens inside each fold.obs_date and then randomly shuffled into train/test: future observations train a model evaluated on the past, and the same customer can appear in both sides. Fix: split on a time cutoff, group by customer.refund_count — over what window? If refunds accrue during the 90-day label window, this feature is the label wearing a costume. The defect is not in the code; it is in the column, and only domain interrogation catches it. Ask: for each feature, was every input knowable at prediction time?Four defects, three of which run clean and produce beautiful metrics. That is why the review order is fixed: splits first, preprocessing second, feature timing third, metrics last — ordered by how expensive each is to discover late.
AI-generated tests do not verify AI-generated analysis code — they encode the same misunderstanding of the data. Your added check is always one the generator did not think to write: a control total, a temporal boundary assertion, a known-answer case.
Keep a personal defects.md of every AI-introduced defect you catch, with the prompt that produced it. After a month you will know your assistant's failure distribution on your data domain better than any benchmark does — and your review time drops because you check the top of that list first.
Decide these once, in daylight — not per-artifact at 6 p.m.:
| Artifact | Standing default |
|---|---|
| Throwaway EDA cell, one-off chart | Run it, eyeball against raw data, delete. No review ceremony |
| Analysis feeding a memo or decision | Full read; recompute one headline number independently; numbers pinned in the write-up |
| Generated SQL against the warehouse | Reconciliation total + join audit + EXPLAIN, every time |
| Model training / evaluation code | The big-five review in fixed order; splits re-derived by hand |
| Anything merging to a shared repo or pipeline | All of the above, plus tests you wrote, plus provenance disclosed in the PR |
The refactor from analysis_final_v3 (2).ipynb to a tested package used to cost a week; it is now an afternoon of agentic work plus your review — and it is the workflow where agentic tools (Claude Code, Cursor's agent mode) genuinely earn their keep, because the task is multi-file, mechanical in the middle, and verifiable at the edges.
1. Freeze behavior first. Before any restructuring, capture what the notebook currently produces: run it top to bottom on a pinned input and save the outputs (model metrics, key aggregates, a prediction sample) as golden files. This is your regression harness — without it, "the refactor works" is a vibe.
2. Brief the agent with structure, not vibes. A refactor prompt that produces reviewable work:
Refactor churn_analysis.ipynb into a package:
- src/churn/{data.py,features.py,train.py,evaluate.py}, typed,
with a CLI entry point (train + evaluate commands).
- Pure functions; all I/O at the edges; config via a dataclass.
- pytest suite: unit tests for every feature function with
hand-computed expected values, plus a regression test that
reproduces golden_metrics.json within 1e-6.
- pandera schema for the input DataFrame: dtypes, nullability,
value ranges taken from the profiling findings in profile.md.
- Do NOT change any computation. If you find a computation that
looks wrong, add a TODO comment and tell me — do not fix it.
Work in steps; show me the module layout before writing code.
The last instruction is load-bearing twice over. "Do not fix what looks wrong" prevents the agent from silently "improving" a computation and breaking golden-file equivalence; and agents do find real bugs during refactors — you want them surfaced as findings, reviewed by you, and fixed in a separate commit with the golden files consciously regenerated.
3. Gate the review at the seams. You do not re-read every mechanically moved line. You review four things:
4. Data validation is part of the contract. The profiling findings from the top of this module become executable:
import pandera.pandas as pa
schema = pa.DataFrameSchema({
"customer_id": pa.Column(str, unique=False, nullable=False),
"monthly_spend": pa.Column(float, pa.Check.ge(0)),
"last_30d_logins": pa.Column(int, pa.Check.in_range(0, 5000)),
"obs_date": pa.Column("datetime64[ns]",
pa.Check.le(pd.Timestamp.now())),
})
validated = schema.validate(df, lazy=True) # collect all failures
Great Expectations fills the same role at pipeline scale; pandera wins for package-internal contracts. Either way, the principle is the one this whole module runs on: constraints the AI helped you discover become constraints the AI-refactored code must obey, checked by machinery rather than memory.
5. CI closes the loop. Tests, schema validation, and the golden-file regression run on every PR. Coinbase-style mandates — AI-assisted engineering with mandatory human review — are spreading into data organizations; the CI harness is what makes "a third of this package was AI-written" a neutral fact instead of a risk disclosure.
Review the agent's tests hardest. The characteristic failure is circular testing — computing the expected value with the same function under test, which asserts only that the code equals itself. Every expected value in a unit test should be derivable by hand on a napkin.
Vendor-neutral picks, two to three per job, all substitutable:
ydata-profiling for the sweep; pandera or Great Expectations for contracts.pytest; nothing here is vendor-locked.Free path (India-aware): Colab and Kaggle notebooks cost nothing and cover every exercise in this module — no GPU is needed for any of it. Databricks Community Edition is free for the warehouse-assistant workflow. Assistant free tiers are adequate at this module's volumes; the entire module is designed to run at ₹0 API spend, and nothing later in the track assumes you paid for anything here.
Your first 15 minutes Monday: the next time an assistant hands you a train-and-evaluate script, don't read it top to bottom — jump straight to the split, then the preprocessing, then each feature's time-of-knowledge, then the metric, in that fixed order, before you ever look at the reported number. That is Neha's six-minute falsification, and it is the whole module in one habit.
Check yourself: Karthik's AI-generated churn script runs clean and reports 0.94 accuracy. Why is "it runs clean" a reason to worry more, not less — and what is the first thing you inspect?
This module's companion playbooks package the reusable assets: the profiling-prompt and findings-to-schema recipe, the EDA hypothesis fan-out template, the SQL reconciliation checklist, the big-five defect review card with the fixed review order, the standing verification-defaults table for adaptation to your team, and the agentic refactor briefing template with the golden-file harness procedure. Do this at work this week → pb-01 Rapid Dataset Profiling and Data-Quality Report with an AI Assistant (generate the profiling code, not the profile), pb-08 Model Debugging with AI: Error Slicing, Learning Curves, and Leakage Hunting (Neha's defect hunt as a repeatable card), and pb-09 Notebook to Production Package: Agentic Refactor with Review Gates (the afternoon refactor with golden-file gates). Two graded labs anchor the module: L1 is a timed planted-defect review — AI-generated analysis scripts seeded with leakage, split, and metric defects, scored on catch rate and false-alarm rate; L2 is the notebook-to-production refactor, graded on golden-file equivalence, test quality (hand-computed expected values), and the quality of your review notes at the seams.