Information in this post reflects publicly available sources as of October 21, 2026.
A team switches its support-ticket tagger to a reasoning model after it tops their internal benchmark. A week later, tagging takes nine seconds instead of one, the bill has quadrupled, and the tags are no more accurate than before. In the same week, an engineer hands that model a date-range bug that the fast model had confidently misdiagnosed three times. It finds the off-by-one in a single attempt.
Both results come from the same feature. Reasoning models spend extra tokens thinking before they answer. On a problem with several dependent steps, that thinking is the difference between right and wrong. On a problem with one obvious step, it is pure cost.
Every major provider now ships models that reason, and most expose a dial for how much. This post explains what the dial actually controls, what you pay for, and how to decide where to set it.
The problem: answering in one pass
A plain language model writes its answer one token at a time, and every token is final the moment it is written. It cannot sketch an approach, notice a mistake halfway through, and start again. Whatever reasoning happens must happen inside the computation for each token, which is plenty for "translate this sentence" and not enough for "find the scheduling conflict across these four calendars."
Engineers discovered early that asking a model to "think step by step" improved its answers on hard problems. Writing intermediate steps gave the model room to work: each step became context for the next. This technique is called chain-of-thought prompting, and it was the first sign that spending more tokens at answer time could buy accuracy.
Reasoning models make that behavior native. Instead of relying on a prompt to coax out steps, they are trained to produce long internal reasoning before answering, and to use it well. The general idea of buying better answers with more computation at answer time is called test-time compute.
How it works
Thinking before answering
A reasoning model's response has two parts. First come thinking tokens (OpenAI calls them reasoning tokens), where the model works through the problem. Then comes the final answer. Providers generally keep the raw thinking hidden and show you the answer, plus an optional summary of the thinking if you ask for one.
The thinking is not decoration. In it, the model breaks the problem down, tries an approach, checks intermediate results, and backs up when something does not fit. You will often see this in summaries as phrases like "wait, that contradicts the earlier constraint," followed by a revised plan.
The economics follow from the tokens post in this series. Thinking tokens are generated like output, so they are billed at output rates, take up room in the context window, count against the output limit, and take time to decode at the same speed as the answer. A request that produces 300 visible tokens may have generated 4,000.
Trained, not just prompted
What separates a reasoning model from a regular model told to think aloud is training. Reasoning models are trained with reinforcement learning on problems whose answers can be checked automatically, such as math with known solutions and code with test suites. The model is rewarded for reaching correct answers, and over many rounds it learns which thinking habits help: decomposing problems, verifying steps, and abandoning dead ends.
This explains where reasoning helps most. The gains are largest on tasks that resemble that training: math, logic, code, and multi-step analysis with a definite right answer. On open-ended writing or simple recall, the benefit is small, because there is nothing to work out.
It also explains a practical prompting change. Instructions such as "think step by step" or detailed scripts of how to reason add little for these models and can even get in the way. Provider guidance converges on the same advice: state the goal, the constraints, and the output format clearly, define what "done" means, and let the model plan its own steps.
The effort dial
Most reasoning models expose a setting that controls how much they think. OpenAI and Anthropic call it effort, with levels such as low, medium, and high. Google calls it thinkingLevel on Gemini 3 and later models, while Gemini 2.5 uses thinkingBudget, an explicit token count.
Effort is a guide, not a meter. Modern models adapt to the request: at the same effort level, a trivial question gets little or no thinking and a hard one gets a lot. Anthropic calls this adaptive thinking, where the model decides per request whether to think at all. Raising effort makes the model more willing to think and to think longer; it does not guarantee a particular token count.
The relationship between effort and quality has a familiar shape. On hard tasks, accuracy rises with effort and then flattens. On easy tasks, it barely moves. Cost and latency, meanwhile, keep climbing, so the last step up the dial is usually the most expensive improvement you can buy.
In code
Here is a call to OpenAI's Responses API with effort set explicitly, followed by the two checks that matter most: whether the response finished, and how many tokens went to thinking.
from openai import OpenAI
client = OpenAI()
resp = client.responses.create(
model="gpt-5.5", # any reasoning-capable model
reasoning={"effort": "low"}, # supported values vary by model
max_output_tokens=25_000, # caps thinking AND the visible answer
input="Which of these three refund requests violates the 30-day policy? ...",
)
if resp.status == "incomplete":
# The limit was reached, possibly mid-thinking, before any answer appeared
print("Incomplete:", resp.incomplete_details.reason)
else:
print(resp.output_text)
usage = resp.usage
print("thinking:", usage.output_tokens_details.reasoning_tokens,
"of", usage.output_tokens, "output tokens")The output limit covers thinking and answer together, so a limit sized for the answer alone can be consumed entirely by thinking, leaving an incomplete response with no visible text. OpenAI suggests reserving at least 25,000 tokens when you start and tuning down from observed usage. The usage fields tell you how much of each response went to thinking, which is the number to watch in production. Anthropic and Google report the same breakdown under their own field names.
Thinking between tool calls
In an agent, reasoning does not only happen at the start. Current models can think between tool calls, reading each result and deciding what to do next. This interleaved thinking is a large part of why reasoning models are better at multi-step agent work: a surprising tool result gets considered rather than blindly passed along.
It comes with a protocol obligation. When a reasoning model calls a tool, its response carries reasoning state alongside the call: reasoning items in OpenAI's API, thinking blocks in Anthropic's, and encrypted thought signatures in Gemini's. You must send that state back, unmodified, with the tool results. Drop it, and the model loses the reasoning behind its own tool call; some APIs reject the request outright.
When to use it, and when not
The question to ask of any task is whether it needs several dependent steps, and whether there is a right answer to reach. Debugging, multi-constraint planning, reconciling contradictory documents, math, and the planning steps of an agent all qualify. These are the tasks where a fast model sounds confident and gets it wrong.
Many tasks do not qualify. Classification, extraction, short factual answers, rewriting, and formatting are mostly one step of pattern matching, and the extra thinking adds cost without adding accuracy. Real-time chat and voice have a latency budget that long thinking simply breaks.
Since most applications mix both kinds of work, the strongest pattern is to choose effort per task rather than per application. Route with rules where task types are known, such as low effort for tagging and high effort for root-cause analysis. Where they are not known, a small classifier or a cheap model can decide, as described in the cost control post.
Make the choice with an eval rather than intuition. Run your own task set at two or three effort levels and compare accuracy, tokens, and latency side by side, as in the evals post. The answer is often that a lower effort level loses nothing, and occasionally that a hard task needs more than you expected.
What goes wrong
Empty answers. An output limit sized for a short answer gets eaten by thinking, and the response ends before any visible text appears. Check the stop reason on every call, size limits for thinking plus answer, and lower the effort if you cannot afford the room.
Overthinking. Reasoning models can spend thousands of tokens on questions that need none. A 2024 study found early open reasoning models used about 20 times as many tokens as conventional models to answer "what is 2 plus 3," mostly re-verifying an answer they already had. Adaptive thinking has reduced this, but low effort on simple routes is still the reliable fix.
Treating the thinking as an explanation. It is tempting to show users the reasoning, or to audit decisions by reading it. Research from Anthropic in 2025 found that when models were given a hint that changed their answer, they mentioned it in their reasoning only about 25% of the time for Claude 3.7 Sonnet and 39% for DeepSeek R1. The thinking is useful for debugging, but it is not a faithful record of why the model decided what it did.
Broken tool loops. Agents that rebuild message history from scratch, or that strip unfamiliar content blocks, drop the reasoning state that must accompany tool results. The symptoms range from API errors to an agent that seems to forget its plan after every tool call. Pass reasoning items, thinking blocks, and signatures back exactly as received.
Cache misses from changing effort. Effort settings are part of the request that providers cache. Changing effort mid-conversation can invalidate the cached prefix, which raises cost and latency on every following turn. Pick one setting per conversation where you can; some providers now offer a way to change effort between turns without losing the cache, but those features are new and model-specific.
Latency surprises in the interface. A user staring at a blank screen for twenty seconds assumes the product is broken. Stream thinking summaries or progress indicators so the wait is visible, and put hard time limits on interactive routes.
Cost, latency, and complexity
The arithmetic is simple and lopsided. Take a 2,000-token prompt and a 300-token answer, at an illustrative $3 per million input tokens and $15 per million output tokens. Without thinking, the call costs about $0.0105. Add 4,000 thinking tokens and it costs about $0.07, almost seven times as much, because every thinking token is billed at the output rate.
Latency moves even more. At about 100 tokens per second, decoding the 300-token answer takes three seconds. Decoding 4,300 tokens takes about 43. That gap is invisible in a benchmark score and very visible to a user waiting in a chat window.
Against that, a reasoning model can replace machinery. A single high-effort call sometimes beats a hand-built chain of prompts, a retry loop, and a verification step, with less code to maintain. The fair comparison is the reasoning call against the whole pipeline it replaces, measured on accuracy, cost, and time, not a reasoning call against a single fast call.
Context is the third cost. Thinking tokens occupy the window during the call, and some models keep earlier thinking in context across turns, which makes long conversations grow faster. Track thinking tokens per conversation alongside the rest of the budget, as described in the observability post.
The current landscape
Every major provider offers reasoning, with different controls and names. Supported levels vary from model to model even within one provider, so check the documentation for the exact model you call.
| Provider | Control | What you see |
|---|---|---|
| OpenAI | reasoning.effort, with levels from none to max; each model supports a subset | Optional reasoning summaries; reasoning items to pass back in tool loops, encrypted when you do not store responses |
| Anthropic | Adaptive thinking with effort from low to max; fixed budget_tokens is the legacy mode | Summarized or omitted thinking; thinking blocks to pass back unmodified with tool results |
thinkingLevel on Gemini 3 and later; thinkingBudget in tokens on Gemini 2.5 | Optional thought summaries; thought signatures required with function calls | |
| Open-weight | Model-specific, such as the thinking and non-thinking modes in Qwen3 or the effort levels in gpt-oss | Usually the full reasoning text, since you run the model yourself |
Several things are still moving. OpenAI offers a separate pro reasoning mode on some recent models that does more work per request at standard token rates. Anthropic's per-message effort changes are in beta, and OpenAI's equivalent mid-conversation update is limited to specific model families. Defaults also differ: some models default to medium, others to high, and some cannot turn thinking off at all.
Open-weight reasoning models, including the DeepSeek, Qwen, Kimi, and GLM families and OpenAI's gpt-oss, make the trade-offs visible in a different way. Running them yourself, you see every thinking token and pay for it in GPU time instead of output pricing. The same rule applies: thinking is only worth it where there is something to work out.
Summary
A reasoning model is a regular model that has learned to spend tokens working on a problem before it answers. That makes the right question not "is this model smarter?" but "does this task have steps to work through?" Where it does, thinking buys accuracy that no amount of prompting a fast model will match. Where it does not, it buys only cost and delay.
So treat effort as a per-task setting, chosen with an eval, rather than a model upgrade applied everywhere. Size output limits for the thinking you cannot see, pass reasoning state back intact in tool loops, and do not mistake the thinking for an honest explanation. The best reasoning setup is usually a mix: a little thinking on most requests, and a lot on the few that need it.
Part of the Applied AI series: AI from the builder's side.