Lab 02 · Baseline Bake-Off: Classical vs LLM Classifier

Both classifiers are scored on the same frozen eval slice, and the report closes with a one-paragraph recommendation. The skill is not either model; it is the discipline of making the comparison fair and the economics explicit.

Time & tier

~2 hours hands-on · Core tier · API spend ≤ $0.50 with a cheap-tier model.

Setup

Python 3.10+ with scikit-learn, numpy, matplotlib, and the openai SDK (used as a universal client — point base_url at OpenAI, Gemini's OpenAI-compatible endpoint, or Groq's free tier). Data comes from scikit-learn's bundled 20 Newsgroups fetcher (a stable public source; ~14 MB one-time download), four confusable categories, headers/footers stripped so the task is honest.

"""l2_baseline.py — classical baselines for the bake-off. Deterministic."""
import numpy as np
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.decomposition import TruncatedSVD
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import f1_score
from sklearn.pipeline import make_pipeline

CATS = ["sci.med", "sci.space", "rec.autos", "comp.graphics"]
train = fetch_20newsgroups(subset="train", categories=CATS, random_state=42,
                           remove=("headers", "footers", "quotes"))
test = fetch_20newsgroups(subset="test", categories=CATS, random_state=42,
                          remove=("headers", "footers", "quotes"))

rng = np.random.default_rng(42)
test_idx = rng.choice(len(test.data), 300, replace=False)  # the shared eval slice
Xte = [test.data[i] for i in test_idx]
yte = test.target[test_idx]

logreg = make_pipeline(
    TfidfVectorizer(sublinear_tf=True, min_df=3, ngram_range=(1, 2)),
    LogisticRegression(max_iter=2000, C=5))
logreg.fit(train.data, train.target)
f1_lr = f1_score(yte, logreg.predict(Xte), average="macro")

gbm = make_pipeline(
    TfidfVectorizer(sublinear_tf=True, min_df=3),
    TruncatedSVD(n_components=200, random_state=42),
    HistGradientBoostingClassifier(random_state=42))
gbm.fit(train.data, train.target)
f1_gbm = f1_score(yte, gbm.predict(Xte), average="macro")

print(f"train={len(train.data)} eval_slice={len(Xte)}")
print(f"macro-F1 logreg={f1_lr:.3f}  gbm={f1_gbm:.3f}")
np.save("eval_idx.npy", test_idx)  # reuse the exact slice for the LLM arm

Expected on this seed: macro-F1 logreg=0.829 gbm=0.761 in under a minute of training. Save eval_idx.npy — every arm is judged on the identical 300 documents.

Guided steps

1. 1Run the classical arms above. Note wall-clock training time and that inference cost is effectively zero (~microseconds/doc on CPU). The linear model beating the GBM on sparse text is itself a finding worth one sentence in your report.
2. 2Build the LLM arm. Zero-shot first:

"""llm_arm.py — zero/few-shot LLM classifier on the frozen slice."""
import numpy as np
from openai import OpenAI
from sklearn.datasets import fetch_20newsgroups
from sklearn.metrics import f1_score

client = OpenAI()          # or base_url + api_key for Gemini/Groq-compatible endpoints
MODEL = "gpt-4o-mini"      # any cheap-tier model; record what you used
CATS = ["sci.med", "sci.space", "rec.autos", "comp.graphics"]
test = fetch_20newsgroups(subset="test", categories=CATS, random_state=42,
                          remove=("headers", "footers", "quotes"))
LABELS = list(test.target_names)          # alphabetical; indices match .target
idx = np.load("eval_idx.npy")

def classify(text, shots=""):
    r = client.chat.completions.create(
        model=MODEL, temperature=0, max_tokens=8,
        messages=[{"role": "user", "content":
                   f"Classify the post into exactly one of: {', '.join(LABELS)}.\n"
                   f"{shots}Reply with the label only.\n\nPOST:\n{text[:2000]}"}])
    out = r.choices[0].message.content.strip().lower()
    usage = r.usage
    pred = next((i for i, l in enumerate(LABELS) if l in out), -1)
    return pred, usage.prompt_tokens, usage.completion_tokens

preds, tin, tout = [], 0, 0
for i in idx:
    p, a, b = classify(test.data[i])
    preds.append(p); tin += a; tout += b
y = test.target[idx]
print("zero-shot macro-F1:", round(f1_score(y, preds, average="macro"), 3))
print("tokens in/out:", tin, tout)

3. 1Add the few-shot arm. Build a shots string with two short training examples per class (truncate each to ~400 characters, label them explicitly), rerun, and log tokens again. Track unparseable replies (pred == -1) — that is your schema-validity preview for Lab 03.
4. 2Do the economics. From logged tokens, compute $/1k predictions at your provider's current prices; put classical arms at their real serving cost (approximately $0 marginal). Add a latency column: median seconds/prediction for the LLM vs the sklearn pipeline.
5. 3Plot the frontier. One matplotlib scatter: x = $/1k predictions (log scale), y = macro-F1, one point per arm (logreg, GBM, LLM zero-shot, LLM few-shot). Annotate each point with latency.
6. 4Write the recommendation (yourself, ≤150 words): which arm ships for this task at 1M predictions/month, and what evidence would flip the decision (label scarcity, drifting categories, need for rationale text).

Verification

• •
  All arms scored on the same 300 indices from eval_idx.npy — assert it in code, not by promise.
• •
  Classical results reproduce: logreg ≈ 0.83, GBM ≈ 0.76 macro-F1 (same seed, same versions).
• •
  LLM arms report macro-F1 and token counts and unparsed-reply count; typical cheap-tier zero-shot lands within roughly ±0.10 of the logreg score — if you see 0.4, your label parsing is broken, not the model.
• •
  Total spend for ~600 LLM calls stays under $0.50; the frontier chart has four annotated points.

Completion artifact

bakeoff-report.md with the frontier chart, a four-row results table (F1, $/1k, median latency, failure count), and the recommendation. Self-grading rubric (pass = 8/10):

• Fairness (0–3): frozen slice enforced in code; identical label space; temperature 0 recorded.
• Economics (0–3): $/1k derived from measured tokens, not vendor marketing; latency measured.
• Analysis (0–2): recommendation names the crossover condition, not just the winner.
• Reproducibility (0–2): someone with your two scripts and an API key gets your table.

Stuck?

• LLM replies drift off-label ("This is about medicine"). Tighten to "Reply with the label only", lower max_tokens, and keep the substring matcher — and record the failure rate rather than silently retrying.
• Rate limits on free tiers. Add a 1–2s sleep or batch overnight; 300 calls fits every free tier.
• Different classical numbers. Check sklearn version and that remove=("headers","footers","quotes") is present — leaving headers in inflates F1 by memorizing newsgroup names.

Client-safe variant

Swap 20 Newsgroups for your client's ticket/review corpus only inside the client's approved tenancy (Azure OpenAI/Bedrock), and report the same four-point frontier. The frozen-slice protocol matters more with client data: business stakeholders will remember the single number you show them, so make sure every arm earned it on the same rows. If no API is approved, the LLM arm can run on a local model (Ollama + an 8B instruct model) — the frontier then compares quality against GPU-hosting cost instead of tokens.