Untracked experiments are the technical debt of data science: three weeks later nobody can say which config produced the model in staging.
Tier: Foundation · Workflow area: Experimentation & reporting
This playbook uses an assistant to generate a complete Optuna + MLflow (or W&B) harness against a written experiment spec, then closes the loop the way governance actually demands: model cards and experiment reports drafted automatically from tracked runs — grounded in logged artifacts, never in the model's memory of what you probably did.
Skip heavy search when the baseline is within noise of the ceiling — tuning a hopeless feature set is procrastination.
Write the experiment spec; make the assistant implement it exactly:
Generate an Optuna study over my existing build_pipeline(params) function
(HistGradientBoostingClassifier). Spec:
- Search space: learning_rate loguniform 0.01-0.3; max_leaf_nodes {15,31,63,
127}; min_samples_leaf int 10-200; l2_regularization loguniform 1e-4-1.0.
- Objective: mean PR-AUC over the existing TimeSeriesSplit folds (reuse my
cv object; do NOT create a shuffled split).
- 60 trials, MedianPruner after fold 2, fixed sampler seed 42.
- Each trial = one nested MLflow run logging: params, per-fold and mean
metrics, fit time, data snapshot hash, git commit, pip freeze.
- Parent run logs: search space as JSON, best trial, and the final refit
model as an artifact with signature + input example.
Review for the three classic harness bugs before launching: objective computed on a different split than the baseline (numbers no longer comparable); pruner keying on a metric other than the objective; refit-on-full-train missing at the end (best CV params ≠ a trained artifact).
Run, watch, stop honestly. Launch the study; monitor the optimization history plot. If the best value plateaus by trial ~30, stop — ask the assistant to add a callback that halts after N non-improving trials so the budget decision is enforced by code, not willpower.
Compare against baseline with uncertainty. Pull best-trial fold scores and baseline fold scores; a paired comparison across folds (even eyeballing mean ± std overlap) decides whether the tuned model is genuinely better. A +0.003 mean with overlapping stds is not a win to announce.
Auto-draft the experiment report from the tracking API — not from memory:
Here is JSON from mlflow.search_runs() for study churn_hgb_v3 (all trials:
params, metrics, tags) and the parent-run metadata. Draft an experiment
report with sections: objective & search space; best config vs baseline
(table, mean ± std per metric); parameter-importance summary; what was NOT
explored; reproduction command. Use ONLY values present in the JSON; where
information is missing, write "not logged" rather than guessing.
The "not logged" instruction is the anti-hallucination valve — and its output doubles as a checklist of what your harness should start logging.
Draft the model card the same way. Feed the registered model's run data plus your evaluation-slice results into a model-card template (intended use, training data description, metrics overall and by key segments, limitations, ethical considerations). You write the intended-use and limitations judgments; the assistant assembles the evidence sections from logs.
Register and link. Register the refit model in the MLflow registry; the report and card become run artifacts. The chain dataset-hash → run → model → card is now navigable by anyone.
Reports drafted from search_runs() output are reproducible documentation; reports drafted from "summarize my experiment" are historical fiction with confident formatting.
wandb.log), and reports draft from api.runs() exports.plot_param_importances, not narrative intuition.