Skip to content

AI6 min read

Most of the time, the cheaper model will do

ARQON EngineeringARQON Singapore

Stop sending routine requests to your top model and inference costs fall by around 40%. Few savings are this unglamorous.

The usual pattern goes like this. Pick one model, wire everything to it, then get a shock from the bill. The first instinct is to negotiate rates or trim prompts, but the big win is usually structural. Most requests reaching a production system aren't hard, and they don't need the most capable model.

Sort your traffic first

Before optimising anything, collect a week of real requests and sort them by difficulty. Every system we've measured has had a heavily skewed distribution. The genuinely hard cases are a thin, long tail. The bulk is classification, extraction and formatting, and small models handle those almost indistinguishably from large ones.

Route, then verify

  • Send the easy majority to a small, cheap model.
  • Escalate to a larger model on low confidence, a failed schema check or an explicit refusal.
  • Run both models against your eval and compare. This is what evals are for.
  • Alert on the escalation rate. When your traffic changes, the bill moves before your metrics do.

On the underwriting project, model routing cut inference costs by 41%, with no measurable change in quality on the held-out data. It took four days. Nothing about it was clever, and that is rather the point.

Filed under AI · 2026.03.19

Back to the journal

(→)Keep reading

More from the journal.

AI

· 7 min

Build the eval before the feature

The fastest way to lose three months on an AI project is to ship the demo first and work out whether it's right later.

Integration

· 6 min

Your webhooks will arrive twice

Any integration works on the happy path. Whether you can forget it exists depends on how it behaves on a bad day.

Practice

· 5 min

The handover is the product

If your team can't ship on its own three months after we leave, we've failed, whatever else we delivered.