Information in this post reflects publicly available sources as of October 18, 2026.
Three things happen to the same engineer in one week. A 300-page contract pasted into a chat model is rejected as too long, even though the model advertises a "huge" context window. The monthly API bill reports 40 million tokens for what felt like a few thousand questions. And a frontier model, capable of explaining tax law, miscounts the letter r in "strawberry."
These look like three unrelated surprises. They are one fact showing up three times: language models do not read characters or words. They read tokens, and every limit, price, and delay in an AI application is measured in them.
This is the foundation the rest of this series has been quietly standing on. The cost math in the cost control post, the budgets in the context engineering post, and the step limits in the agents post all count tokens. This post explains what they are and how to reason about them.
Models do not see your text
A model works with numbers, not letters. Before your text reaches it, a tokenizer chops the text into pieces from a fixed vocabulary, typically somewhere between about 100,000 and 250,000 entries, and replaces each piece with its ID. The model reads that sequence of IDs and writes its answer as another sequence, which the tokenizer turns back into text.
The vocabulary is learned from huge amounts of text, mostly with a method called byte-pair encoding, which keeps merging the most frequent character combinations until the vocabulary is full. Frequent words end up as single tokens. Rare words, names, typos, and numbers get split into several pieces.
How it works
What a token looks like
A few patterns show up in almost every tokenizer. The space before a word is usually part of that word's token, so " refund" and "refund" are different tokens. Common English words are one token each. Numbers, dates, identifiers, and unusual words break into fragments.
For English prose, a useful rule of thumb is about four characters per token, or roughly 750 words per 1,000 tokens.
That last example explains the strawberry problem. The model never sees the letters s-t-r-a-w-b-e-r-r-y. It sees three chunks, and it has to infer from training how many r's hide inside them.
Character-level tasks such as counting letters, reversing strings, or checking exact spelling are genuinely awkward for models. When they matter, have the model call code instead.
Numbers have a similar problem. A long number becomes several arbitrary fragments, which is part of why models are unreliable at exact arithmetic on large values. Use a calculator tool or code for anything that must be exactly right, as the structured outputs post suggested for any value your code depends on.
Not every language is equal
Tokenizers are trained mostly on English text, and it shows. A 2023 study of 17 tokenizers found that the same content translated into different languages could take up to 15 times as many tokens, even for tokenizers built to be multilingual. Newer tokenizers have narrowed the gap, but it has not disappeared.
That gap has three practical effects for anyone serving users in more than one language. The same answer costs more, because billing is per token. It takes longer, because generation speed is per token. And less of it fits in the context window.
If your product serves non-English users, measure token counts in their languages rather than assuming the English rule of thumb.
Code and data are not prose
The four-characters rule describes English prose. Code, JSON, logs, and tables behave differently, and usually worse. Indentation, brackets, quotes, and repeated key names all cost tokens, and long identifiers such as customerRefundApprovalTimestamp split into many fragments.
This matters most for tool results and retrieved data. An API response pretty-printed as JSON, with every key repeated on every record, can take several times the tokens of the same facts written as a compact table or a few sentences. Reformatting tool output before it reaches the model is one of the cheapest token savings available, and it often improves answers too, because there is less noise to read past.
Identifiers carry a second risk. Because a long ID or hash becomes a string of arbitrary fragments, a model copying it can drop or swap a piece. When exact values must survive a round trip through the model, keep them in your code and refer to them by a short handle, or validate them on the way out.
Counting tokens
Because every model family has its own tokenizer, the same text produces different token counts on different models. For OpenAI models you can count locally with the open-source tiktoken library. Anthropic and Google provide token-counting API endpoints for their models, and Hugging Face's tokenizers library covers open-weight models.
import tiktoken
enc = tiktoken.get_encoding("o200k_base") # tokenizer used by recent OpenAI models
text = "The customer's refund was approved on 2026-10-03."
tokens = enc.encode(text)
print(len(text.split()), "words,", len(tokens), "tokens")
print([enc.decode([t]) for t in tokens]) # see exactly where the boundaries fall
def fits(prompt_tokens: int, max_output: int, window: int, margin: float = 0.05) -> bool:
"""Leave room for the answer, plus a safety margin for counting differences."""
return prompt_tokens + max_output <= window * (1 - margin)The first lines show where a tokenizer splits your actual text, which is often surprising for code, JSON, and identifiers. The fits check captures the rule most context errors break: the prompt and the answer share the same window.
The context window is a budget
A model's context window is the maximum number of tokens it can handle in one call, counting everything: the system prompt, tool definitions, conversation history, retrieved documents, the user's message, and the answer it writes. Most models also have a separate, smaller output limit, the most tokens they will generate in one response. For reasoning models, the hidden reasoning tokens count toward that output too.
Thinking of the window as a budget makes problems easier to spot. A 300-page contract is roughly 150,000 words, which is about 200,000 tokens of English, before the system prompt or the answer. That is why it failed in the opening story: the window was large, but not large enough for the document plus everything else.
History is the part of the budget that grows. Every turn of a conversation and every step of an agent adds to it, which is why long conversations eventually hit the limit and why the context engineering post spent so much time on trimming and compaction. A bigger window delays that moment; it does not remove it.
Where the time goes
Tokens also explain latency, because a model handles input and output in two very different phases. In prefill, it reads all the input tokens at once, in parallel, which is fast per token. In decode, it writes the output one token at a time, each depending on the last, which is much slower per token.
The time to first token is mostly prefill, so it grows with prompt length; prompt caching shortens it by skipping work on a repeated prefix. After that, the response arrives at the model's decode speed, often somewhere between tens and a couple of hundred tokens per second. Quadruple the answer length, and the wait roughly quadruples.
This is also why streaming feels so much faster. The total time is the same, but showing tokens as they are decoded means the user starts reading after the first fraction of a second instead of waiting for the full answer.
When this matters
Token thinking matters whenever you are estimating cost, setting limits, or diagnosing slowness. Use the four-characters rule for rough English estimates, then measure with the real tokenizer before anything that affects money or limits. Measure separately for code, structured data, and every language you serve, because the rule of thumb is wrong for all three.
Estimating before you build is worth ten minutes. Take a realistic request, add up the system prompt, tool definitions, typical retrieved content, and a long conversation, then multiply by expected volume. That rough sheet usually shows which part of the budget dominates, and it is almost never the part people were worried about.
It matters less for everyday prompt writing. You do not need to choose words by token count, and saving ten tokens in a system prompt is rarely worth the clarity it costs. Spend that attention on what actually dominates: large documents, growing history, verbose tool output, and long answers.
What goes wrong
Answers get cut off. When a response hits the output limit, generation simply stops, often mid-sentence or mid-JSON. The API reports this as a stop reason such as max_tokens or length. Check it on every response, and set output limits with headroom for the longest legitimate answer.
Conversations hit the wall. A chat or agent that works for twenty turns fails on the twenty-first, either with an error or because the application silently drops the oldest messages, including the instructions that mattered. Track token usage per conversation and trim or summarize deliberately before the limit, not after.
Budgets use the wrong unit. Limits set in characters or words, or copied from a different model, drift from reality. A switch to a model with a different tokenizer can change token counts for the same prompt, and with them the cost and the fit. Recount when you change models.
Tool definitions eat the budget quietly. Every tool you give a model is described in its context on every call: name, description, and parameter schema. Connect a few MCP servers with dozens of tools each and the definitions alone can run to tens of thousands of tokens before the user has said anything. Load tools on demand, or expose fewer, broader ones, when the list grows.
Estimates replace measurements. Teams size a feature with the four-characters rule, then discover in production that their JSON-heavy prompts run at two characters per token. Log the actual input and output token counts the API returns on every call, and build budgets from those numbers rather than from estimates.
Big windows hide quality loss. A model that accepts a million tokens does not attend equally well to all of them, as the context engineering post described. Fitting in the window is necessary, not sufficient.
Images and files are tokens too. A screenshot or a page of a PDF is converted into tokens before the model sees it. A single 1280-by-800 screenshot comes to roughly 1,400 tokens under Anthropic's estimate, so a few dozen images can fill a budget faster than pages of text.
Cost and latency in tokens
Put the two phases together with pricing and a useful asymmetry appears. Take a support answer with a 6,000-token prompt and a 300-token reply. At an illustrative $3 per million input tokens and $15 per million output tokens, the input costs $0.018 and the output $0.0045, about $0.0225 in total. Input is 80% of the cost.
The time splits the other way. Prefill takes about 0.4 seconds, and decoding 300 tokens at about 80 tokens per second takes about 3.75 seconds. Output is roughly 90% of the wait. So the main lever for cost is usually input, through caching and trimming, while the main lever for speed is usually output, through shorter answers and streaming.
Reasoning models shift this balance. Their hidden reasoning tokens are billed and generated like output, so a model that thinks for 2,000 tokens before a 300-token answer is slower and costlier than the visible answer suggests. Effort or thinking-budget settings let you limit that where the task does not need it.
The current landscape
Context windows have grown enormously. Frontier models from the major providers now commonly offer windows from around 200,000 tokens to a million or more, and some offer larger windows still. Output limits remain much smaller, typically in the tens of thousands of tokens. Check each model's documentation, because limits differ between models from the same provider and sometimes between API tiers.
Tokenizers keep evolving too. Recent tokenizers use larger vocabularies, which tend to encode non-English text and code more efficiently, so a newer model may use noticeably fewer tokens for the same input. That changes cost comparisons between models: price per token only means something alongside how many tokens your actual workload produces on each.
For counting, tiktoken covers OpenAI's tokenizers, Anthropic and Google expose counting endpoints for their models, and Hugging Face's libraries cover most open-weight models. Most observability platforms, as described in the observability post, record token counts per call automatically.
Summary
Tokens are the unit everything in an AI application is measured in. They decide what fits in a request, what a request costs, and how long it takes. They also explain behaviors that look like stupidity, such as miscounting letters and fumbling long numbers, as consequences of how the model sees text.
So think in tokens when it matters. Budget the context window like any finite resource, leave room for the answer, measure real token counts for your languages and data, and remember the asymmetry: input mostly drives the bill, output mostly drives the wait. Every other technique in this series is, at bottom, a way of spending tokens more wisely.
Part of the Applied AI series: AI from the builder's side.