LLM Features and Classifiers vs Classical Models: The Cost-Quality Bake-Off

Tier: Core · Workflow area: Hybrid Classical + LLM Modeling

This playbook runs both: extract typed features from text at scale, then compare four modeling options on a single cost-quality frontier.

When to use

• •
  You have tabular data with neglected free-text columns (complaint text, notes, descriptions) and a supervised target.
• •
  Stakeholders are asking "should we just use an LLM for this classifier?" and you need a defensible answer with a cost axis.
• •
  You maintain a classical text classifier and want to know if LLM features lift it before considering full replacement.

Prerequisites

• •
  A labeled dataset (≥1k rows for a meaningful comparison; stratified sample is fine).
• •
  scikit-learn, xgboost (or LightGBM), pydantic, an LLM API key, and the structured-extraction function from pb-14.
• •
  Per-1M-token pricing for at least two model tiers (e.g. a frontier tier and a cheap tier) — the frontier is meaningless without real prices.

The workflow

1. 1.

   Fix the evaluation contract first. One held-out test split, one metric (macro-F1 or PR-AUC for imbalanced targets), and a cost-per-1k-predictions column in every result table. Freeze the split before any LLM call so no option gets tuned on test data.

2. 2.

   Contender A — classical baseline. TF-IDF + linear model and TF-IDF + XGBoost. This takes 20 minutes and is the bar everything else must clear:

   Copy

   from sklearn.pipeline import make_pipeline
   from sklearn.feature_extraction.text import TfidfVectorizer
   from sklearn.linear_model import LogisticRegression
   baseline = make_pipeline(TfidfVectorizer(ngram_range=(1,2), min_df=3),
                            LogisticRegression(max_iter=1000, class_weight="balanced"))
   

3. 3.

   Contender B — LLM features → GBM. Define a feature schema and batch-extract it over the text column (use the Batch API from pb-16 for anything above a few thousand rows):

   Copy

   class TextFeatures(BaseModel):
       sentiment: Literal["negative", "neutral", "positive"]
       mentions_competitor: bool
       urgency_1to5: int
       topics: list[Literal["billing", "quality", "delivery", "support", "other"]]
   

   One-hot/ordinal-encode these alongside your existing tabular features and train XGBoost. Extract features for train and test with the same model, prompt, and schema version.

4. 4.

   Contender C — zero/few-shot LLM classifier. Direct classification via structured output (Literal over your label set), zero-shot and with 8–16 few-shot examples selected per-class. Run it on the cheap tier and the frontier tier — the delta is often smaller than the price gap.

5. 5.

   Contender D — small fine-tuned encoder (optional stretch). A DistilBERT/ModernBERT-class model fine-tuned on your training set; the strongest classical-adjacent contender when you have ≥5k labels. Fits free Kaggle GPU hours.

6. 6.

   Plot the frontier. One scatter: x = cost per 1k predictions (include amortized extraction cost for B and training cost for D), y = test metric, one point per contender/tier. Add inference-latency as point labels. The decision usually falls out immediately.

Tool variants

• Anthropic: structured extraction and classification via client.messages.parse(..., output_format=Model); use a cheap tier (e.g. claude-haiku-4-5) vs a frontier tier (e.g. claude-opus-4-8) as your two price points — tier routing is the subject here, so measure both.
• OpenAI:client.responses.parse(..., text_format=Model); mirror the same two-tier comparison.
• Open-weights via free inference tiers (Groq/OpenRouter/Gemini free): add a near-zero-cost point to the frontier — often decisive for high-volume classifiers.
• sklearn/XGBoost side: identical regardless of provider; keep the modeling code provider-agnostic so only the feature-extraction call swaps.

Verification

• •
  All contenders evaluated on the identical frozen test split; no LLM prompt was edited after seeing test scores.
• •
  Feature-extraction determinism spot-check: re-extract 50 rows and confirm ≥95% field-level agreement; unstable features are leakage of randomness into your model.
• •
  Cost numbers reconstructed from actual usage token counts in API responses, not guesses.
• •
  Feature importances for Contender B inspected — if no LLM feature ranks in the top 10, the lift is probably noise.

Pitfalls

• Label leakage through the prompt. If your few-shot examples come from the test split, or the extraction prompt names the target ("does this indicate churn?" is the label, not a feature), your LLM numbers are fiction. Extract features, not the target, for Contender B.
• Comparing best-effort LLM vs untuned baseline. Grid-search the TF-IDF/GBM baseline with the same effort you spend prompt-iterating. Asymmetric effort is the most common way these bake-offs mislead.
• Ignoring latency class. A batch-scored nightly model and a 2-second-per-call online LLM are different products; annotate the frontier with latency so nobody ships the wrong quadrant.
• Per-row extraction of a static column. Cache extracted features keyed by text hash; re-extracting on every training run multiplies cost for zero benefit.
• Treating the answer as permanent. Cheap-tier quality and prices move quarterly; keep the harness runnable so the bake-off can be re-run in an hour.

Restricted-environment variant

• •
  Replace hosted extraction with a local 7–8B instruct model + Outlines (pb-14 Layer 3) for feature extraction; Contenders A and D are already fully local.
• •
  For the LLM-classifier contender, use the same local model zero/few-shot — the frontier then compares local-GPU cost (spot T4/A10 in ap-south-1, or free Kaggle hours) rather than API dollars.
• •
  No raw client text in any hosted prompt; if a hosted cheap tier is allowed for benchmarking only, run it on a public or synthetic stand-in corpus and note the caveat in the report.

Success criteria

• •
  A single frontier chart with ≥4 points (baseline, LLM-features+GBM, LLM classifier on two tiers), confidence intervals, and real per-1k costs.
• •
  A one-paragraph recommendation naming the winner for a stated volume and latency requirement, not in the abstract.
• •
  The extraction schema, prompts, split indices, and harness are committed so the bake-off reruns end-to-end with one command.
• •
  At least one surprising finding documented — typically either "LLM features lift GBM by X points" or "the cheap tier matches the frontier tier within noise."