AI6 min read
Most of the time, the cheaper model will do
ARQON EngineeringARQON Singapore
Stop sending routine requests to your top model and inference costs fall by around 40%. Few savings are this unglamorous.
The usual pattern goes like this. Pick one model, wire everything to it, then get a shock from the bill. The first instinct is to negotiate rates or trim prompts, but the big win is usually structural. Most requests reaching a production system aren't hard, and they don't need the most capable model.
Sort your traffic first
Before optimising anything, collect a week of real requests and sort them by difficulty. Every system we've measured has had a heavily skewed distribution. The genuinely hard cases are a thin, long tail. The bulk is classification, extraction and formatting, and small models handle those almost indistinguishably from large ones.
Route, then verify
- Send the easy majority to a small, cheap model.
- Escalate to a larger model on low confidence, a failed schema check or an explicit refusal.
- Run both models against your eval and compare. This is what evals are for.
- Alert on the escalation rate. When your traffic changes, the bill moves before your metrics do.
On the underwriting project, model routing cut inference costs by 41%, with no measurable change in quality on the held-out data. It took four days. Nothing about it was clever, and that is rather the point.
Filed under AI · 2026.03.19
Back to the journal