Feature-Engineering Ideation and Implementation with Leakage Review

Tier: Foundation · Workflow area: Feature engineering

This playbook runs the few-shot ideation → codegen → leakage-review loop on top of a frozen baseline (pb-05).

When to use

• •
  A validated baseline exists and you want systematic lift, not architecture roulette.
• •
  You have relational/temporal raw data (events, transactions, logs) far richer than the current feature table.
• •
  You inherited a feature set and suspect it underuses the domain.

Skip heavy ideation for pure deep-learning-on-raw-signal problems where learned representations replace manual features.

Prerequisites

• •
  A frozen, reproducible baseline metric (pb-05) — without it, "the new features helped" is unfalsifiable.
• •
  pandas (and SQL for warehouse-side features); optionally featuretools for relational aggregation.
• •
  A schema-and-timeline document: for every source table, its grain and when each field becomes known. This document is what makes leakage review possible.

The workflow

1. 1.

   Few-shot ideation, seeded with your best examples. Show the model what good looks like in your domain:

   Copy

   Task: churn model, snapshot-based (features as of snapshot_date, target =
   churn within 90 days). Sources: orders (order-item grain), support_tickets,
   app_events (user-day grain).
   Two features that already work well:
   - days_since_last_order (recency)
   - tickets_30d / tenure_months (normalized friction)
   Generate 25 candidate features across: recency/frequency/monetary, trends
   (30d vs 90d deltas), volatility, lifecycle ratios, cross-source interactions.
   For EACH: name, one-line rationale, source tables, and the exact time window
   relative to snapshot_date. Exclude anything needing data after snapshot.
   

2. 2.

   Cull on paper. Kill duplicates, uncomputable ones, and features whose rationale is circular ("users who churn have churn-like behavior"). Keep 8–12. Your domain sense is the filter — the model over-generates by design.

3. 3.

   Generate implementation with explicit as-of semantics:

   Copy

   Implement these 10 features as a function
   build_features(snapshots: df, orders: df, tickets: df) -> df with one row
   per (customer_id, snapshot_date). Rules:
   - Every aggregation filtered to event_ts < snapshot_date (strict).
   - Use merge_asof / windowed groupbys, no row loops.
   - Missing history → explicit NaN, not 0 (imputation happens in the pipeline).
   - Return also a dict feature_docs: {name: {window, sources, formula}}.
   

4. 4.

   Leakage-review each feature against the timeline doc. For every feature ask three questions: Could any input be recorded after snapshot (late-arriving events, mutable statuses)? Does it embed the target (e.g., account_status that flips when churn is logged)? Does it cross entities (aggregates computed over train+test populations, target encodings fit on full data)? Have the assistant do a first pass — Review each feature for temporal leakage given: orders can be updated up to 7 days after creation; ticket status mutates; app_events arrive with up to 48h delay — then verify its answers yourself.

5. 5.

   Handle target encoding with CV hygiene. If using target/mean encodings for high-cardinality categoricals, they must be fit within CV folds (sklearn TargetEncoder inside the pipeline, or out-of-fold manually). Never accept generated code that computes a target mean on the full frame and merges it in.

6. 6.

   Evaluate as a batch, then ablate. Add all survivors, re-run the pb-05 evaluation. If the metric improves, ask for an ablation script (drop-one-group importance or permutation importance on the time-based validation split) to see which features carry the lift and which are dead weight — prune the dead weight; every retained feature is a maintenance liability.

7. 7.

   Suspicion pass on winners. Any feature producing a large jump gets the pb-05 treatment: does the lift survive on the final held-out period? Does its importance make domain sense? Sudden +0.05 AUC from one feature is a leak until proven otherwise.

Tool variants

• Chat assistant + notebook: the default; ideation and review in chat, code executed by you.
• Agentic coding tool (Claude Code, Cursor): best when features live in a repo as tested transform modules — it can write the pytest cases for as-of correctness alongside (see pb-10).
• featuretools hybrid: let Deep Feature Synthesis enumerate relational aggregates mechanically with a cutoff_time frame, and use the assistant to interpret/prune its output and add the domain-specific features DFS cannot invent.

Verification

• •
  A synthetic-row test passes: construct a customer with a known event exactly at snapshot_date and one just before/after; features must include the "before" and exclude the "at/after" events.
• •
  Target-shuffle test (pb-05) still collapses to the dummy floor with new features included.
• •
  Lift holds on the untouched final test period, not just CV.
• •
  Every retained feature has an entry in feature_docs with window and formula.
• •
  Target encodings verifiably fit out-of-fold only.

Pitfalls

• Window boundary sloppiness.<= vs < on snapshot_date, and windows in calendar days vs business days — generated code picks arbitrarily; your spec must pin it and your synthetic-row test must catch it.
• Late-arriving data optimism. Training features computed on settled historical data, but production computes them on 48h-delayed events — train/serve skew. Simulate the delay in training if serving will have it.
• Ratio features dividing by zero-ish denominators.spend / tenure_days for day-0 users; demand explicit handling, not silent inf → NaN → imputed.
• Keeping every feature that "doesn't hurt." Un-ablated feature sets rot; prune to features that demonstrably carry lift.
• Ideating without grain awareness. Aggregating order-item rows as if they were orders double-counts everything — carry the pb-03 grain discipline here.

Restricted-environment variant

• •
  Ideation needs schema and two example features, not data — safe with hosted models virtually everywhere. Implementation runs locally.
• •
  Air-gapped: local code models handle step 3 acceptably given the strict spec; keep a house library of as-of aggregation utilities so generation is thin wrappers, not novel logic.
• •
  If feature semantics reveal business strategy (e.g., risk signals), alias names in prompts and maintain the mapping locally.

Success criteria

• •
  8+ candidate features implemented and leakage-reviewed in a session that previously produced 2–3.
• •
  Measured, ablation-attributed lift over the frozen baseline on the held-out period — or a clean negative result that saved weeks.
• •
  The synthetic-row as-of test exists and passes in CI.
• •
  feature_docs lets a teammate recompute any feature without reading the code.