Information in this post reflects publicly available sources as of November 1, 2026.
A hospital's data team wants to summarize clinical notes with a language model, and the compliance review ends the conversation quickly: patient records cannot be sent to an outside API. Down the hall, a startup's finance lead notices that one batch-classification job now costs more each month than the engineer who wrote it. A developer on a long flight wants a coding assistant that works without Wi-Fi.
All three end up looking at the same option: a model whose weights they can download and run themselves. Open-weight models have gone from a hobbyist curiosity to a serious alternative. The strongest now trail the best closed models by a small margin on many tasks, and several are free for commercial use.
Running a model yourself trades one set of problems for another, though. This post covers what open weights actually give you, what it takes to run them, and when the trade is worth it.
The problem: an API means someone else's terms
Calling a hosted model is convenient because someone else handles everything: hardware, scaling, updates, and uptime. The cost of that convenience is dependence. Your data goes to the provider, your bill follows their prices, your behavior changes when they update or retire a model, and your product stops when their service does.
For most applications, that is a fine deal. For some it is not: regulated data that cannot leave your network, workloads at a volume where per-token pricing hurts, products that must work offline or on a device, and teams that need a model to behave identically for years. Open weights are the answer for those cases, and only those cases.
How it works
What "open" actually means
An open-weight model is a set of files containing the trained parameters, plus the code to run them. You can download them, run them on your own hardware, and usually fine-tune them. That is different from open source in the strict sense, which would also include the training data and training code. Most "open" models release the weights only.
The license matters more than the label. Some models use permissive licenses: DeepSeek's recent models use MIT, and OpenAI's gpt-oss and Google's Gemma 4 use Apache 2.0.
Others carry conditions. Meta's Llama 4 license requires attribution and a separate license for companies with more than 700 million monthly users, and some Qwen and Mistral models use community or research licenses that restrict certain commercial uses. Terms can differ between versions of the same family, so read the license file of the exact model you deploy.
Three ways to run the same weights
"Running it yourself" covers three quite different setups. The model file can be identical in all three; what changes is who operates it, where the data goes, and how you pay.
Locally, tools such as Ollama, LM Studio, and llama.cpp run small models on a laptop or desktop, including Apple Silicon Macs. This is ideal for development, offline use, and private personal work. On your own GPUs, inference servers such as vLLM, SGLang, and NVIDIA's TensorRT-LLM serve many users at once, in your data center or cloud account. Through an inference provider, you call open models as a hosted API, from companies such as Together AI, Fireworks AI, and Groq, or from cloud platforms such as Amazon Bedrock and Google Cloud's Vertex AI.
The third option is easy to overlook. It keeps the model choice portable, since several providers host the same weights, while avoiding any hardware work. You give up the data and uptime benefits, but you gain the ability to move to self-hosting later without changing models.
Will it fit?
The first practical question is memory. A model's weights take roughly its parameter count times the bytes used per parameter. At 16-bit precision, each parameter takes two bytes, so a 70-billion-parameter model needs about 140 GB just for weights, which is more than one data center GPU holds.
Quantization shrinks that by storing parameters with fewer bits. At 8 bits the same model needs about 70 GB, and at 4 bits about 35 to 40 GB. The quality cost at 8 bits is usually negligible; at 4 bits it is often small but task-dependent, so evaluate the exact quantized file you plan to run. Common formats include GGUF for llama.cpp and Ollama, AWQ and GPTQ for GPU servers, and the 4-bit MXFP4 format that gpt-oss ships in.
Weights are not the whole bill. While generating, the server keeps a KV cache, the model's working memory for every token in every active request. It grows with context length and with the number of concurrent users, and for long contexts it can rival the weights themselves. Budget for it, or the model that loaded fine in testing runs out of memory under real traffic.
Mixture-of-experts changes the math
Many leading open models use a mixture-of-experts (MoE) design. The model contains many specialist sub-networks, and only a few run for each token. gpt-oss-120b, for example, has about 117 billion parameters in total but only about 5 billion active per token.
The consequence is a split between memory and speed. All the parameters must fit in memory, because any expert might be needed. But each token only computes through the active ones, so the model runs at roughly the speed of a much smaller model. That is why gpt-oss-120b fits in 80 GB and runs quickly on one data center GPU, and its 21-billion-parameter sibling runs within 16 GB on a well-equipped laptop.
Small models go further than you expect
The headline open models are huge, but much of the practical value sits at the other end. Models of a few billion to a few tens of billions of parameters are often trained partly on the outputs of much larger ones, a technique called distillation, and they inherit a surprising share of the larger model's skill on everyday tasks.
For narrow, well-defined work, such as classifying tickets, extracting fields, redacting personal data, or routing requests, a small open model can match a frontier model while running on a single modest GPU, a laptop, or even a phone. Start your evaluation with the smallest candidate that might work, and move up only when your eval says it falls short.
The same API shape
Switching to open weights rarely means rewriting your application. Ollama, vLLM, LM Studio, and most inference providers expose an OpenAI-compatible endpoint, so the client code you already have mostly works.
# Start a local server first, for example:
# ollama pull gpt-oss:20b (laptop, port 11434)
# vllm serve openai/gpt-oss-20b (GPU server, port 8000)
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="unused")
resp = client.chat.completions.create(
model="gpt-oss:20b",
messages=[{"role": "user", "content": "Summarize this incident report in three bullet points: ..."}],
)
print(resp.choices[0].message.content)Only the base URL and model name change. That makes it cheap to run the same eval against a hosted model and a local one, which is exactly what the choosing a model post recommends before any switch. Compatibility is not perfect, though. Tool calling, structured outputs, and reasoning settings vary between servers and models, so test the features you depend on rather than assuming they carry over.
When to use it, and when not
Open weights earn their keep in four situations. The first is data that cannot leave your control, such as health records, privileged legal material, or classified work.
The second is high, steady volume of a well-defined task, where per-token pricing adds up. The third is offline or on-device use, from field equipment to laptops on planes. The fourth is long-term stability: a model you host never changes or retires unless you decide it should.
They are a poor fit when you need the very best model for hard, open-ended tasks, where the leading closed models still tend to have an edge. They are also a poor fit when volume is low or spiky, because GPUs bill by the hour whether or not requests arrive. And they are a poor fit for teams without anyone to own serving: a model server is production infrastructure, with on-call, upgrades, and capacity planning.
Fine-tuning is a related reason teams choose open weights. You can adapt an open model with your own data and keep the result, and small fine-tuned open models often match much larger general ones on narrow tasks. The fine-tuning post covers when that is worth doing at all.
What goes wrong
The benchmark model is not the model you run. Published scores are for full-precision weights with the vendor's recommended settings. Your 4-bit quantization, different chat template, or shorter context can perform noticeably worse. Evaluate the exact file, server, and settings you will deploy.
Chat templates and settings. Each model expects its prompts in a specific format, and many expect particular sampling settings. A wrong template produces subtly worse answers rather than errors, which makes it hard to notice. Use the template shipped with the model, and check the model card's recommended settings.
Memory surprises under load. A model that answers one test request fine runs out of memory with twenty concurrent users and long documents. Load-test with realistic context lengths and concurrency before launch, and cap both in the server configuration.
License surprises. A model is chosen for its scores, and its license turns out to forbid the intended use, or to require attribution the product team never planned for. Have the license reviewed as part of model selection, not after launch.
Security is now yours. Model files can contain malicious code in some formats, so prefer safe formats such as safetensors and download from the publisher's official repository. Patch your serving stack, and remember that safety tuning in open models can be weaker or removed, so the guardrails described in the guardrails post matter even more.
Cost, latency, and complexity
Self-hosting replaces a variable cost with a fixed one. Whether that saves money depends almost entirely on how busy you keep the hardware.
Take an illustrative case: one GPU at about $2,200 a month, serving a mid-size MoE model at up to 7.8 billion tokens a month, compared with a hosted API at $0.50 per million tokens. On hardware cost alone, self-hosting breaks even around 4.4 billion tokens a month. Add $2,000 a month of engineering time for operations, a conservative figure, and the break-even moves to about 12.8 billion tokens. Below that, the API is cheaper, and spiky traffic pushes the point higher still.
Latency can go either way. A dedicated server close to your users, with no shared queue, can respond faster than a busy public API. An under-provisioned one, or a large model on modest hardware, can be much slower. Specialized inference providers often deliver the fastest output speeds of all, because their whole business is optimizing that.
Complexity is the cost teams underestimate. Running models means choosing GPUs, configuring servers, monitoring memory and throughput, upgrading as better models arrive, and handling failures at night. Locally, for one developer, that is trivial. As a production service, it is a platform someone must own.
The current landscape
The open-weight field moves fast, and rankings change monthly. Leading families include DeepSeek, Qwen from Alibaba, Kimi from Moonshot AI, GLM from Z.ai, MiniMax, and NVIDIA's Nemotron, alongside Meta's Llama, Google's Gemma, Mistral, and OpenAI's gpt-oss. Many of the strongest recent releases are large MoE models with hundreds of billions of total parameters, built for data center hardware, while each family typically also offers small models that run on a laptop.
For running them, Ollama and LM Studio are the simplest starting points locally, built on engines such as llama.cpp, with Apple's MLX popular on Macs. For serving at scale, vLLM and SGLang are the common open-source choices. Hugging Face remains the main distribution hub, and independent trackers such as Artificial Analysis compare open and closed models on quality, speed, and price.
Because releases arrive so quickly, treat any ranking, including this one, as a starting point. The model that tops a leaderboard this month may be superseded next month, and its license may differ from its predecessor's. Your own eval and a license review remain the deciding steps.
Summary
Open weights do not make AI free. They change who pays for what: you trade per-token prices and someone else's terms for hardware, operations, and full control. The model itself is often close to the best available; the real question is whether you want to be in the business of running it.
So start from the reason. If data must stay in-house, the model must work offline, or it must never change, open weights may be the only answer. If the reason is cost, do the arithmetic with honest utilization and engineering time included. And remember the middle path: hosted open models give you portability now, with the option to bring them in-house when the numbers say so.
Part of the Applied AI series: AI from the builder's side.