Skip to content

AI7 min read

Build the eval before the feature

ARQON EngineeringARQON Singapore

The fastest way to lose three months on an AI project is to ship the demo first and work out whether it's right later.

When we're called into a stalled AI project, we usually find the same scene. There is a prototype, and in a demo it works beautifully. Someone in leadership has seen it and is asking when it can launch. And nobody in the room can answer a very simple question: is it right?

Right, here, doesn't mean right in general. It means right on the thousand inputs that matter to this business, checked in a way that two people measuring separately would reach the same conclusion. Without that, every decision after it is made on a hunch. Hunches don't survive production traffic.

What an eval is

An eval is not a benchmark score. It is a dataset gathered from real work, a scoring function that reflects what users actually care about, and the ability to run the two on every change. That is the whole idea. The hard part isn't the tooling. It is deciding what "good" means and writing it down.

  • A dataset drawn from real inputs, including the awkward ones people currently handle by hand.
  • Ground-truth labels from the people who make that call today: underwriters, clinicians, support leads.
  • A scoring function that can tell "wrong" from "unhelpful", because the fixes are completely different.
  • A held-out set nobody tunes against, so you can tell real improvement from overfitting.

"I don't know" is a feature

A system that can't hold back an answer ends up guessing. On an underwriting project for a US insurer, the biggest jump in user trust didn't come from better retrieval. It came when we let the assistant say "I don't know", and scored whether it said so when it should have. Once underwriters believed a confident answer had a reason to be confident, they stopped re-checking every single one.

For the first three weeks, what they built was a way to tell us whether we were wrong. Looking back, that was the project.
— VP Engineering at a US insurer

What changes

With an eval in place, arguments get shorter. A model upgrade stops being a leap of faith and becomes something you measure and decide. Cost cutting stops being frightening, because you can prove nothing degraded when you route the routine 80% of traffic to a cheaper model. And when leadership asks whether it's ready, you can answer with a number.

It takes a week or two up front. We have never regretted spending that time. We have been called in many times to clean up projects that didn't.

Filed under AI · 2026.06.18

Back to the journal

(→)Keep reading

More from the journal.

AI

· 6 min

Most of the time, the cheaper model will do

Stop sending routine requests to your top model and inference costs fall by around 40%. Few savings are this unglamorous.

Integration

· 6 min

Your webhooks will arrive twice

Any integration works on the happy path. Whether you can forget it exists depends on how it behaves on a bad day.

Practice

· 5 min

The handover is the product

If your team can't ship on its own three months after we leave, we've failed, whatever else we delivered.