Baseline Model Scaffolding: Problem Framing to Evaluated Pipeline in Minutes

Tier: Foundation · Workflow area: Modeling

The discipline is writing the spec before the code and running a leakage checklist after, because generated pipelines fail exactly where review is boring.

When to use

• •
  Starting any supervised problem: you need the reference number everything else must beat.
• •
  Evaluating whether a problem is worth pursuing ("is there signal here at all?").
• •
  Replacing a metric-free legacy heuristic with something measurable.

Skip codegen when your team has a house pipeline template — extend that instead of generating a parallel structure.

Prerequisites

• •
  Python with scikit-learn (plus XGBoost/LightGBM if tabular boosting is your default), and a fixed random seed policy.
• •
  An AI coding assistant (Claude Code, Cursor, or chat + notebook).
• •
  The dataset profiled (pb-01) and the target variable's construction understood — when is it known, relative to the features?

The workflow

1. 1.

   Write the spec first, in the prompt. The spec forces the decisions leakage hides in:

   Copy

   Build a baseline sklearn pipeline. Spec:
   - Task: binary classification, target = churned_90d (defined at day 90
     after snapshot_date; features must use only data available AT snapshot).
   - Data: pandas df, 180k rows; numeric: [tenure_days, mrr, tickets_30d],
     categorical: [plan, country, channel]; missing values present in mrr.
   - Split: time-based — train on snapshots <= 2025-09, test on 2025-10..12.
     No shuffled split; explain why in a comment.
   - Pipeline: ColumnTransformer (median impute + scale numeric; most_frequent
     impute + OneHotEncoder(handle_unknown="ignore") categorical) →
     LogisticRegression AND HistGradientBoostingClassifier as two candidates.
   - Eval: PR-AUC and ROC-AUC via time-series CV on train; single final test
     evaluation. Include DummyClassifier(strategy="prior") as the floor.
   - Everything inside Pipeline/ColumnTransformer — nothing fit outside CV.
   

2. 2.

   Review the generated code against the leakage checklist before running:

   • •
     All transforms inside the Pipeline (no fit_transform on the full frame)?
   • •
     Split respects time, and grouped entities (GroupKFold if one customer spans rows)?
   • •
     No feature derived from post-outcome information (tickets_30d counted before snapshot, not before churn)?
   • •
     Encoders configured for unseen categories at inference?
3. 3.

   Run and demand the floor comparison. A baseline is meaningful only against the dummy: PR-AUC of 0.31 means little until you see the prior is 0.08. If your fancy candidate barely beats dummy, that finding is the deliverable.

4. 4.

   Ask for a suspicion report on its own output:

   Copy

   Act as a reviewer of the pipeline you just wrote. List the top 5 ways this
   baseline could be overstating performance (leakage, split, metric, target
   definition), each with a one-line test to check it.
   

   Run the cheap tests. A classic: shuffle the target and re-run — CV score should collapse to the dummy's; if not, something leaks.

5. 5.

   Probe feature legitimacy. Pull feature importances / coefficients and interrogate the top ones: mrr_last_invoice dominating a churn model may mean invoices stop before churn is recorded — a target proxy. Ask the assistant for the "too good to be true" analysis of each top feature given your schema timeline.

6. 6.

   Freeze and record. Save the pipeline (joblib), the metric table (dummy vs both candidates, CV mean ± std, final test), seed, and data snapshot hash. This is the number all future work must beat; make it reproducible (see pb-07 for tracking).

Tool variants

• Agentic tools (Claude Code, Cursor): hand over the spec and let them create module + test files and run them; review diffs rather than transcribing.
• Chat + notebook: paste spec, copy code; fine for single-file baselines.
• AutoML as a cross-check (not the workflow): a quick FLAML/AutoGluon run tells you whether your generated baseline is in the right league; a large gap means your pipeline or their validation is broken — find out which.

Verification

• •
  Target-shuffle test collapses performance to the dummy floor.
• •
  Test metric is within plausible range of CV (a big jump upward on test screams leakage or a broken split).
• •
  Retraining with a different seed moves metrics within CV std.
• •
  Top features survive the timeline interrogation (nothing computable only after the outcome).
• •
  Rerunning the saved script end-to-end reproduces the metric table.

Pitfalls

• Shuffled splits on temporal data. The single most common generated-code error in this task; it inflates every metric and invalidates the baseline. The spec must state the split; step 2 must verify it.
• Preprocessing outside the pipeline.StandardScaler().fit_transform(X) before the split looks harmless and leaks distribution information into CV.
• Metric mismatch. Accuracy on an 8% positive class, or ROC-AUC when the business cares about precision at fixed alerts — pin metrics in the spec.
• Baseline gold-plating. Assistants add SMOTE, stacking, and threshold tuning unprompted. Strip it; the baseline's job is to be simple, honest, and hard to beat accidentally.
• One CV number. Always mean ± std across folds; a single-fold number invites decision-making on noise.

Restricted-environment variant

• •
  The prompt needs schema and the spec, never rows — this workflow is hosted-assistant-safe in most orgs as written; generated code runs locally.
• •
  Air-gapped: keep a local spec-template file; a local code model (Ollama + a code-tuned model) fills it adequately because the spec carries the intelligence. Review discipline is unchanged.
• •
  If even column semantics are sensitive, alias names in the spec and un-alias in the generated code locally.

Success criteria

• •
  Spec to evaluated two-candidate baseline with dummy floor in under 30 minutes.
• •
  All verification checks pass, including the target-shuffle test.
• •
  The recorded baseline is reproducible from a clean environment by a colleague.
• •
  You caught at least one would-be leak (split, transform placement, or feature timeline) during review rather than after deployment.