Map Model Tiers and Routing to a Cost-Quality Frontier

Workflow area: AI Product Economics · Tier: Advanced · Hands-on time: ~45 minutes

Model choice is a recurring product decision disguised as a one-time technical one. Prices shift quarterly, new tiers appear monthly, and the difference between a frontier model and a small one on your specific task is frequently smaller than the leaderboard gap — or occasionally larger, in the opposite direction. Teams that never measure this either overpay 10× for indistinguishable quality or quietly ship a cheap model that fails the cases that matter. The frontier chart replaces both mistakes with a picture.

When to use — and when not to

• Use it when the feature has meaningful volumeat thousands of requests a day, a 10× per-token difference is a real gross-margin line; below that, pick a good model and revisit later.
• Use it when requests vary in difficultymixed traffic (most queries easy, some hard) is exactly where routing pays; uniform-difficulty traffic just needs the right single tier.
• Use it at every provider price change or new model releasethe frontier is a living chart; re-running the suite is an hour once this is set up.
• Skip routing for latency-critical uniform tasksan escalation hop adds a round trip; if p95 matters more than cost, pick one tier that clears the bar.
• Skip it without an eval seta frontier drawn from vibes is a scatter plot of opinions. pb-40's golden set is the prerequisite.

The recipe

1. 1.

   Define the quality floor per traffic slice, not per feature. Split your golden set by difficulty or intent: e.g. "reformat/extract" cases, "summarize" cases, "reason across documents" cases. For each slice, state the minimum acceptable score (from your launch bar). The frontier is only honest sliced — a model can ace easy traffic and fail hard traffic while averaging fine.

2. 2.

   Run the same suite across three-plus tiers. In promptfoo, list candidate models as multiple providers over the identical test set — a frontier tier, a mid tier, a small/cheap tier (add an open-weight option via OpenRouter if it's a candidate). One config, one command, a matrix of scores. Record per-slice pass rates and per-request token costs from the run stats.

3. 3.

   Draw the frontier and find the dominated options. One chart: x-axis cost per 1,000 requests, y-axis eval score (per slice, one chart each or small multiples). Any model below-and-right of another is dominated — more money for less quality on your task. Ask Claude to narrate it:

   Copy

   Here are eval results for one feature across model tiers.
   Slices and launch bars: [e.g. extract ≥95%, summarize ≥90%,
   multi-doc reasoning ≥85%]
   Results (per slice: pass rate, mean cost/request):
   [PASTE YOUR MATRIX]
   Traffic mix: [e.g. 70% extract, 25% summarize, 5% multi-doc]
   
   1. Identify dominated models (worse quality at higher cost).
   2. For each slice, name the cheapest model clearing its bar.
   3. Compute blended cost/1k requests for: (a) frontier-model-
      everywhere, (b) cheapest-clearing-model per slice, (c) cheap
      default + escalate to frontier on the hard slice only.
   4. State what routing signal would be needed for (c) and its risk.
   

   Sample output:

   Copy

   (a) frontier everywhere: $41.20/1k requests, all bars cleared
   (b) per-slice cheapest: $9.80/1k, all bars cleared
   (c) cheap default + escalation on multi-doc: $7.10/1k IF the
   router catches ≥95% of multi-doc intents; each missed escalation
   is a below-bar answer on your hardest slice. Risk: router
   misclassification is invisible without logging escalation
   decisions as an eval-able event.
   

4. 4.

   Specify the routing policy as product behavior. If (c) wins, write the half-page spec: the routing signal (intent classifier, input length, user tier, explicit "think harder" affordance), the fallback rule (on classifier uncertainty, escalate — pay for doubt), what gets logged (route decision per request, so misroutes become eval cases), and the UX question — do users see or choose the escalation? A visible "deep analysis" mode can turn a cost optimization into a feature.

5. 5.

   Wire the re-run trigger. LiteLLM or a gateway gives you per-model cost telemetry in production; put a quarterly calendar item — or a trigger on any provider price change over 20% — to re-run the suite and redraw the chart. Yesterday's frontier is not evidence.

Make it yours

• Latency frontier overlay: add p95 latency as point size on the chart — some "cheaper" options lose on the latency bar, and the visual settles it.
• Prompt-per-tier variant: small models often need a more explicit prompt to clear the bar; test each tier with a tuned prompt rather than one shared prompt before declaring it below-bar — sometimes $2/1k plus a better prompt beats $40/1k.
• Cascade evaluation: test the "try cheap, verify, retry with frontier on failure" cascade — total cost includes the doubled calls on retries, so measure the real blended rate on your traffic mix, not the optimistic one.

Quality check

• Scores come from your eval setnot a leaderboard, not the provider's benchmark page; the chart's axis says "our golden set vN."
• Slices are separateda blended average hiding a failed hard-slice bar is the classic way this analysis lies.
• Costs use measured tokensfrom real run stats, including system prompt and retrieval context, not the request-body guess.
• Routing misses are counted as quality failuresoption (c)'s score includes router error, or it's fiction.
• The chart has a date and a re-run triggerfrontier charts expire; an undated one is a future bad decision.