Every few weeks, one of the big labs releases a model that tops the benchmarks. The announcement is convincing, the examples are impressive, and someone on the team asks whether you should switch.

Maybe. A model release is a candidate for testing, not an automatic upgrade.

Benchmarks measure someone else's tasks. Your product has its own: your customers' documents, your prompts, your tools, your edge cases. A model can be better on average and worse at the one thing your customers rely on.

When I build, I pick the model for each task from Anthropic, OpenAI and Google. Changing the model behind a feature customers use is a different decision, because it changes what they see. Here is how I approach it.

Build a small test set from real tasks

Anthropic's guide to evals for AI agents suggests that 20 to 50 simple tasks drawn from real failures is a great start, and advises against waiting for a perfect suite.

That is a manageable amount of work. I pull tasks from:

  • The cases the current model gets wrong.
  • Typical requests, so you know the everyday experience holds up.
  • Edge cases that caused support tickets or confused users.
  • Anything with legal, financial or safety weight in your domain.

For each task, write down what a good result looks like. Not a perfect answer word for word, but what must be true for it to count as right.

Compare what actually matters

Running the same test set on the current model and the candidate gives you a fair comparison. The guide notes that latency, token usage, cost per task and error rates can all be tracked on a fixed bank of tasks, alongside whether the answer was right.

So compare:

  • Correctness. Did it get the right answer?
  • Completion. Did it finish the task, or stop, refuse or loop?
  • Latency. How long did people wait?
  • Cost. What did each successful result cost, including retries? I wrote about why that number matters in what an AI feature costs to operate.

Model outputs vary from run to run, so the guide recommends running multiple trials rather than trusting one pass. A model that gets something right once is not the same as a model that gets it right every time.

Look for regressions, not just improvements

The guide separates two kinds of evaluation. Capability evals ask what an agent can do well, and are expected to start with a low pass rate. Regression evals ask whether the agent still handles everything it used to, and should pass nearly 100% of the time.

Model switches are where regressions hide. The new model might write longer answers, format output differently, call tools in a different order, refuse requests the old one handled or follow your instructions a little less literally. None of that shows up in a benchmark. All of it shows up for customers.

Scores only tell you so much, which is why the guide's advice to read the transcripts is worth following. A short session reading what the new model actually did often tells you more than the summary table.

Automated tests are one layer

The guide lists several ways to understand how an agent performs: automated evals, production monitoring, A/B testing, user feedback, manual transcript review and systematic human studies. It compares them to the Swiss cheese model from safety engineering, because no single layer catches every issue.

In practice, a test set gets you to "probably safe." Rolling out to a small share of usage, watching monitoring, reading real transcripts and listening to users gets you the rest of the way.

Keep a rollback path

Switching should be reversible in minutes, not days. A few habits make that possible:

  • Keep the model choice in configuration, not scattered through the code.
  • Pin specific model versions rather than an alias that can change underneath you.
  • Version prompts with the model they were tuned for, because a prompt that works well on one model may need changes on another.
  • Keep the old configuration ready until the new one has proven itself in production.

Sometimes you will not get a choice, because providers retire older models. A test set and a rollback path turn a forced switch from an emergency into a routine. The guide makes a similar point: teams with evals can upgrade to a new model in days, while teams without them face weeks of testing.

Decide whether it matters to users

The last question is the one people skip. Even if the new model is measurably better, is it better at something your users will notice?

If it improves a score nobody feels, while costing more or responding more slowly, staying put is a perfectly good decision. If it fixes the failures your customers complain about, switch, carefully.

A better model is only an upgrade if it is better at your product.

A switching checklist

  1. Run your test set on both models, more than once.
  2. Compare correctness, completion, latency and cost per successful result.
  3. Check every regression case, and read a sample of transcripts.
  4. Roll out to a small share of real usage and watch it.
  5. Keep the old configuration ready to switch back.
  6. Ask whether the improvement is one users will feel.

If you are weighing a model switch and do not have a test set yet, email me at kevin@kevinhorek.com. A first one can be small.