Choosing an AI model: three questions before the leaderboard

Comparisons of GPT, Claude, Gemini and Llama change every few months, and none of them answers the question that matters inside a company: which model works with its documents, its volume and its data constraints. The choice comes down to three concrete questions. And the architecture has to let the answer change without rebuilding the system.

A public leaderboard answers a different question from the one the company has

The benchmarks published by providers and labs measure general capabilities: reasoning, coding, long-document comprehension. They are useful for knowing which model family is current. They say little about whether a model correctly classifies support tickets written in a hurry, with in-house abbreviations and half-scanned attachments.

That is why the first step is building an evaluation set of your own. Fifty to a hundred real examples from the process are enough, each with the correct answer noted by someone from the business: emails to classify, invoices to extract fields from, questions customer service actually receives. With that set, two or three models can be tested in an afternoon. The result is a number that can be compared.

In practice, that set becomes the regression test for the whole system. Every change of prompt, model or provider is checked against it before reaching production, the same way a code change is.

Where the data may go settles half the decision

The second question is about governance. Some processes handle data that can travel to a provider's API, and some handle information that must stay inside the company's infrastructure: financial records, customer histories, documents covered by contract.

For the first case, commercial models over an API (Anthropic, OpenAI, Google) are the fastest route and the best quality per euro. For the second there are two paths. Azure and AWS serve those same models inside the customer's own account and region, through Azure OpenAI Service or Amazon Bedrock, so the data never leaves an environment the company already administers. And open-weight models such as Llama or Mistral can be deployed on the company's own servers when the requirement is stricter still.

What changes with a self-hosted model is the operating cost. Someone has to size the GPU, roll out new versions and watch latency. It is worth taking on only when the data requirement demands it or the volume pays for it.

Volume turns the price per token into an architecture decision

At a few hundred requests a day, the model's price barely registers on the bill and the sensible choice is the best model available. At tens of thousands of requests a day, the gap between a small model and a large one is the gap between an affordable cost and one that stops the project.

The usual answer is routing. A small, fast model classifies and resolves the simple cases. Cases that fail a validation check or cross a complexity threshold go to a large model. Whatever neither resolves with enough confidence goes to a person. The same pattern serves customer support, document extraction and contract review.

In return, it has to be measured. Routing only works if the evaluation set from the first question exists, because that is what says which share of cases can stay on the cheap model without losing quality. Two further levers cut the bill without touching quality: the prompt caching providers offer for prefixes repeated in every request, and batch processing for anything that does not need an immediate answer.

The architecture must allow a model swap without a rewrite

The models at the top of today's comparisons will be replaced within months. That is why no process should call a provider directly from the business logic. In the systems we build, an intermediate layer receives the request, picks the provider and model for that type of task, records the cost and returns the answer in the same format whichever model sits behind it.

That layer does three things for the company. It lets a new model be tested against the evaluation set and switched on with a configuration change. It gives a fallback provider for when the main one fails or retires a model, which happens more often than it seems. And it gives a view of cost per process that the provider's invoice never gives.

Put another way, the important decision is reversible. Choosing Claude, GPT or a Llama deployed on Azure today matters less than being able to change it within a week when the evaluation set shows another model doing the job better or cheaper.

Where to start this week

• Pick a single process with measurable volume and a business owner who can annotate correct answers.
• Collect 50 real examples with their expected answer and keep them as the evaluation set.
• Sort that process's data into three groups: can go to an API, must stay in the company's own cloud, cannot leave the company at all.
• Test two models from different providers against the set and record quality, latency and cost per thousand requests.
• Design the integration with an intermediate layer from day one, even if only one model sits behind it.

If the case fits, we review it in a first 60-minute call.