Information in this post reflects publicly available sources as of November 1, 2026.
A team picks the model at the top of a popular leaderboard for its invoice extraction service. On their invoices, it does slightly worse than the mid-tier model it replaced, at three times the price. Six months later, after they have finally tuned their prompts for it, the provider announces its retirement with 60 days' notice.
None of this is bad luck. It is what happens when a model is chosen like a phone, by reading reviews and buying the top one. Models are closer to hires. The question is not who is best in general but who does this particular job well, at a price you can sustain, and what happens when they leave.
This post covers how to make that choice: what public benchmarks can and cannot tell you, how to measure models on your own work, how to use more than one, and how to survive the constant churn of releases and retirements.
The problem: too many models, the wrong evidence
The choice is wide and keeps moving. Each major vendor offers several tiers, typically a large flagship, a balanced mid-size model, and a small fast one, with reasoning settings on top. Add open-weight families, regional providers, and cloud marketplaces, and there are hundreds of plausible options. New releases arrive monthly.
The evidence most people use is public rankings. Those answer a real question, but not yours. A benchmark measures performance on someone else's tasks, graded someone else's way, often on problems the model may have seen in training. Your workload has its own documents, formats, edge cases, and definition of a correct answer.
How to choose
Narrow the field in stages
Choosing well is a funnel. Each stage is cheaper than the next, so you spend real effort only on a handful of candidates.
Start with hard constraints, which rule models out on facts rather than opinion. Can your data go to this provider, in this region, under these retention terms? Is it available through the cloud you already buy from?
Does it support the features you need, such as tool calling, structured outputs, image input, a large enough context window, prompt caching, or batch pricing? Many choices end here before quality is even discussed.
Public benchmarks then shrink what remains to a shortlist of four to six. Use them to spot obvious mismatches, such as a model that is weak at code when your task is code. Then measure the shortlist on your own work, which is the only stage that actually decides.
What public benchmarks can tell you
Benchmarks have three well-documented problems. The first is saturation: once top models all score in the high 90s, the benchmark stops separating them. The second is contamination: public test questions leak into training data, so a score can reflect memory rather than skill.
In February 2026, OpenAI published an audit explaining why it no longer reports SWE-bench Verified, a widely cited coding benchmark. It concluded that every frontier model it tested had seen some of the problems and solutions in training, and that many of the audited tasks had tests that rejected correct fixes.
The third problem is selection. Arena-style leaderboards, where people vote between two anonymous answers, measure something real: which answers people prefer in casual use. But a 2025 study, "The Leaderboard Illusion," found that some providers privately tested many variants and published only the best, with one testing 27 before a single release. Picking the best of many noisy scores inflates the result even when the variants are identical.
None of this makes benchmarks useless. Independent trackers such as Epoch AI and Artificial Analysis publish methodology, measure speed and price directly, and flag benchmarks they consider flawed. Read them for broad capability and for numbers you cannot easily measure yourself, then confirm on your own data.
Measure on your own work
Your own eval does not need to be elaborate. Collect 50 to 200 real examples of the task, including the awkward ones that caused trouble before, each with a known good answer or a clear grading rule. Run every shortlisted model on the same cases with the same prompt, and record three numbers per model: accuracy, cost, and latency. The evals post covers grading in depth.
import time
import litellm # one interface to many providers
MODELS = ["provider-a/small", "provider-a/mid", "provider-b/mid", "provider-c/large"]
def run(model: str, cases: list[dict], grade) -> dict:
correct, cost, latency = 0, 0.0, []
for case in cases:
start = time.perf_counter()
resp = litellm.completion(model=model, messages=case["messages"])
latency.append(time.perf_counter() - start)
cost += litellm.completion_cost(completion_response=resp)
correct += grade(resp.choices[0].message.content, case["expected"])
latency.sort()
return {"model": model,
"accuracy": correct / len(cases),
"cost_per_1k": cost / len(cases) * 1000,
"p95_seconds": latency[int(0.95 * (len(latency) - 1))]}
results = [run(m, cases, grade) for m in MODELS]The model names are placeholders; a gateway library such as LiteLLM uses a provider prefix plus the model ID. The grade function is yours: an exact match for extraction, a schema check for structured output, or a model-based judge for open-ended answers. Run the cases more than once if outputs vary, and look at the failures, not just the totals.
Pick the cheapest model that is good enough
Plot the results and a pattern usually appears. A few models sit on a frontier, where no other model is both cheaper and more accurate. Everything else is dominated and can be dropped.
Then decide what "good enough" means before you look at prices. If 88% accuracy meets the need, with the remainder caught by validation or review, the right model is the cheapest one above that line. The next model up might add three points for five times the cost. Sometimes that is worth it; often it is not, and you should make the call deliberately rather than by default.
Compare cost on your workload, not price per token. Models differ in tokenizer efficiency, verbosity, and how much they think, so a model with a lower per-token price can cost more per task. The harness above measures what you actually pay.
Hosted API or open weights
One choice cuts across every tier: whether to call a hosted model through an API or run an open-weight model whose weights you can download. Hosted frontier models usually lead on the hardest tasks and need no infrastructure. You pay per token and accept the provider's terms, limits, and retirement schedule.
Open-weight models trade that convenience for control. You choose where the data goes, the model never changes unless you change it, and nobody can retire it. For narrow, high-volume tasks, a mid-size open model on your own hardware or a dedicated host can be much cheaper per request once utilization is high.
The catch is that you now own serving, scaling, and upgrades, and GPUs cost money whether or not traffic arrives. Many teams land in the middle: open-weight models bought as a hosted API from an inference provider, which keeps the model choice portable without running the hardware. Evaluate them in the same harness as everything else.
Use more than one model
Most applications contain several tasks, and they rarely share the same best model. Tagging and extraction are often handled perfectly by a small model, while root-cause analysis needs a large one. Assigning a model per task, or per route, is usually the largest single saving available.
A gateway makes this practical. Your application calls one interface, and configuration decides which model answers. Gateways typically add fallbacks, retrying on another provider when one fails or rate-limits you, and they record cost and latency per call for the observability you need anyway.
Routing can be static, with rules mapping known task types to models, or dynamic, with a router predicting per request whether a cheaper model will do. Static routing is predictable and easy to test. Dynamic routing helps when requests are mixed and unlabeled, but it adds a component that can misroute, so evaluate the router like any other model.
Pin versions and plan for retirement
Model names come in two kinds. A pinned ID refers to one fixed model. An alias is a pointer that the provider can move to a newer release. Aliases are convenient for experiments and risky in production, because behavior can change without any change on your side.
Even pinned models do not last forever. Providers publish deprecation schedules: Anthropic, for example, commits to at least 60 days' notice before retiring a publicly released model, and other providers publish their own timelines. Treat a model change like a dependency upgrade: rerun your eval, compare against the current model, and roll out gradually.
Keep your prompts and eval cases in version control next to the model ID they were tuned for. When a retirement notice arrives, the work is then a rerun and a review, not an investigation.
When to switch, and when not
New releases are not by themselves a reason to switch. Re-evaluate when something concrete changes: a deprecation notice, a price cut in a tier you could use, a new capability you need, a quality problem your users report, or a new model that independent measurements place well above your current one on tasks like yours.
Equally, do not wait for a crisis. Running your eval against one or two new candidates each quarter keeps the comparison cheap and keeps you ready to move. A team that can switch models in a day negotiates and adapts very differently from one that needs a quarter.
What goes wrong
Choosing by leaderboard. The top-ranked model is chosen and never measured on the actual task. It may be better, worse, or just more expensive. Without your own eval, nobody knows.
Prompts that do not transfer. A prompt tuned for one model often underperforms on another, which makes the old model look better in a quick comparison. Give each candidate a fair attempt, with modest prompt adjustments, before declaring a winner.
An eval that is too easy. If every candidate scores 98%, the eval cannot separate them and the cheapest model looks free of risk. Include the hard cases, the long inputs, and the formats that broke things before.
Silent drift. An application pinned to an alias changes behavior overnight when the alias moves. Even on pinned IDs, providers can update serving infrastructure. Watch quality metrics in production so changes surface as alerts, not complaints.
One vendor, no exit. Provider-specific features creep into the code, and switching becomes a project. Keep provider calls behind one interface, keep a tested fallback, and check your rate limits before launch day rather than during it.
Cost, latency, and complexity
The savings from matching models to tasks are large. Suppose a service costs $10,000 a month running everything on a large model. If 80% of requests can go to a small model costing a tenth as much, the bill becomes about $2,800, a 72% cut, assuming quality holds on those routes. Latency usually improves too, because small models respond faster.
Latency deserves its own measurement. Providers differ in time to first token and in output speed, and these vary by time of day and region. Measure the 95th percentile, not the average, because that is what users waiting on a slow response notice.
The price of using many models is complexity. Each model means another prompt variant, another column in the eval, another deprecation schedule, and another set of failure modes. Two or three models, chosen deliberately, captures most of the benefit. Beyond that, each extra model should justify itself with measured savings.
The current landscape
Closed models come from OpenAI, Anthropic, Google, and others, each in several tiers with reasoning controls. Open-weight families such as Llama, Qwen, DeepSeek, Mistral, Kimi, GLM, and OpenAI's gpt-oss can be run yourself or bought as a hosted API. Many models are available both from their creator and through cloud platforms such as Amazon Bedrock, Google Cloud's Vertex AI, and Microsoft Foundry, which matters when procurement or data residency dictates where you can buy.
For access and routing, aggregators such as OpenRouter offer many models behind one API. Gateways such as LiteLLM and Bifrost run in your own infrastructure with fallbacks and spend tracking. Dynamic routers include Not Diamond, Azure's model router in Microsoft Foundry, OpenRouter's Auto Router, and the open-source RouteLLM project; Amazon Bedrock also offers prompt routing within a model family.
For evidence, Epoch AI and Artificial Analysis track capabilities, prices, and speeds with published methods, and LMArena publishes preference rankings. Each provider's deprecation page is the authoritative source for retirement dates. Check them for every model ID you run in production.
Summary
There is no best model, only the best model for a task, at a price, under constraints, for now. Public benchmarks are a starting point for a shortlist, weakened by saturation, contamination, and selective reporting. Your own eval, run on your own cases, is the only evidence that decides.
So build the eval once and reuse it. Pick the cheapest model that clears a bar you set in advance, assign models per task behind a single interface, pin versions, and keep a fallback ready. When the next release or retirement notice arrives, changing models becomes a routine rerun rather than a scramble.
Part of the Applied AI series: AI from the builder's side.