Model Debugging with AI: Error Slicing, Learning Curves, and Leakage Hunting

Tier: Core · Workflow area: Model debugging & diagnostics

The same skills double as your defense against AI-generated modeling code: the final step is a review drill on code with planted bugs.

When to use

• •
  Offline metrics fine, production metrics poor (or vice versa).
• •
  A metric moved suspiciously — in either direction — after a change.
• •
  CV looks good but stakeholders report obviously wrong individual predictions.
• •
  You are reviewing AI-generated modeling code and need a systematic bug hunt.

Prerequisites

• •
  The trained pipeline, evaluation data, and per-row predictions accessible in a notebook.
• •
  sklearn diagnostics (learning_curve, permutation_importance, calibration tools) and SHAP for local explanations.
• •
  An assistant that accepts pasted tables/outputs (and images of plots, if multimodal).
• •
  The pb-05 baseline artifacts for reference — debugging without a baseline is astrology.

The workflow

1. 1.

   Build the evidence pack before prompting. Compute: metric by key slices (df.groupby(['segment','month']).apply(pr_auc)), confusion matrix at the operating threshold, learning curve, permutation importance, calibration curve, and the 20 worst false positives and negatives with features. Diagnosis quality is capped by evidence quality.

2. 2.

   Demand a differential, not an answer:

   Copy

   Churn model, PR-AUC 0.41 CV vs 0.27 on Oct-Dec holdout. Artifacts:
   [slice-metric table] [importance table] [learning curve values]
   [calibration table]. List the 6 most likely root causes ranked by
   consistency with THIS evidence. For each: which artifact supports it, which
   contradicts it, and a test costing <10 lines of pandas to discriminate it
   from the others. Do not recommend fixes yet.
   

   The "no fixes yet" rule prevents the classic failure: optimizing before localizing.

3. 3.

   Run the discriminating tests in order of cheapness. Typical differentials and their tests: temporal drift (retrain on rolling windows — does the gap track recency?); leakage in CV (target-shuffle test; grain check for entity overlap across folds); slice concentration (is the drop isolated to one country/segment/month?); label shift (compare positive rates train vs holdout); feature pipeline skew (recompute one feature independently and diff). Paste each result back; make the model update its ranking explicitly.

4. 4.

   Use SHAP for the "wrong individual predictions" class. For the 40 worst errors, get SHAP values and ask the assistant to cluster the error stories: Group these 40 misclassified rows by their dominant SHAP features. Name each error cluster, estimate its share, and say whether it looks like a data problem, feature gap, or genuine irreducible case. Verify cluster claims by reading a few rows yourself.

5. 5.

   Interpret learning curves with discipline. Paste the curve values, not just the picture: converged-and-close means more data will not help (feature or label problem); a large gap means variance (regularize or simplify); both-bad means bias. Make the assistant commit to one regime and one implied next action, then sanity-check against your slice evidence.

6. 6.

   Fix one cause, re-run the full evidence pack. Never bundle fixes: you must know which one worked. Confirm the slice that was broken improved and no other slice regressed.

7. 7.

   Run the adversarial review drill (planted-bug practice). Periodically, review code seeded with the canonical bugs: scaler fit before split, train_test_split(shuffle=True) on temporal data, target encoding on full data, evaluation on the training fold, off-by-one in as-of joins, wrong-threshold metrics. Have one assistant plant 3 bugs in a working pipeline; a fresh session — or you, timed — finds them:

   Copy

   Insert 3 realistic evaluation-inflating bugs into this sklearn pipeline
   (leakage, split, or metric bugs). Output only the modified code — no hints.
   

   Score yourself: found/missed, and time-to-find. This calibrates how much to trust your own review pass in pb-05/pb-06.

Tool variants

• Chat assistant + notebook: default; paste tables as text for exact numbers rather than screenshots.
• Agentic tool (Claude Code): it computes the evidence pack itself; require it to show each artifact before any diagnosis, and treat its conclusions as hypotheses for step 3.
• Multimodal chat: reads residual plots, calibration curves, and SHAP beeswarms directly; confirm any number it reads off a chart against raw values.

Verification

• •
  The accepted root cause is supported by a discriminating test result, not by narrative fit alone.
• •
  The fix moves the previously broken slice and leaves others within noise.
• •
  Post-fix, target-shuffle and grain checks still pass (fixes can introduce leaks too).
• •
  In the review drill, you find at least 2 of 3 planted bugs unaided.
• •
  The diagnosis is written down: symptom → differential → tests → cause → fix, so the next person pattern-matches faster.

Pitfalls

• Vibes-in, vibes-out. Prompting "my model got worse, why?" without artifacts yields a listicle of every possible cause — technically true, diagnostically useless.
• Anchoring on the first plausible story. Models commit early; the forced ranking-update in step 3 (and a fresh thread for a second opinion) counters it.
• Fixing the metric instead of the model. Threshold-tuning a drifted model "back to green" hides the drift; distinguish recalibration from repair.
• SHAP overreach. SHAP explains the model, not the world; a feature with high attribution in errors is a lead, not a cause.
• Trusting the drill-planting session. The planting assistant sometimes introduces 2 bugs or 4, or one that does not inflate metrics — run both versions and confirm the inflation before drilling.

Restricted-environment variant

• •
  Evidence packs are aggregates (slice metrics, curves, importances) — generally shareable with hosted assistants; the 40 worst-error rows are not: replace identifiers, or describe rows as feature-pattern summaries.
• •
  Air-gapped: local models handle the differential-diagnosis format well since the reasoning is over pasted numbers; keep a printed house checklist of the canonical bug list for review drills without any model.
• •
  Regulated settings: log the debugging session (prompts + artifacts + conclusions) into the experiment tracker as evidence of due diligence.

Success criteria

• •
  A real performance regression localized to a verified root cause via discriminating tests, and fixed without collateral regressions.
• •
  Time-to-diagnosis measurably shorter than your last untooled incident.
• •
  Review-drill score of ≥2/3 planted bugs found, improving across drills.
• •
  A written diagnosis artifact exists in the tracker, linked to the affected run.