Information in this post reflects publicly available sources as of November 2, 2026.
A support team tests its new assistant against 200 questions written by the product team. It scores 94%.
In production, the questions look different. Customers type on phones with their thumbs, switch between English and Spanish mid-sentence, paste text copied out of scanned receipts, and forward CSV exports with columns in whatever order their system produced. Satisfaction scores slip, but nobody can point to a single dramatic failure.
Meanwhile, the same engineers use chat models every day with sloppy, typo-filled questions and get excellent answers. So which is it? Are models robust to messy input or not?
Both experiences are real. Models are tolerant of some kinds of mess and quietly brittle to others, and the research is now specific enough to say which is which. This post covers how models cope with typos, bad grammar, mixed languages, OCR and speech noise, and inconsistent formatting, what the measured drops look like, and what actually needs to be clean.
Why models tolerate any noise at all
A model never sees your characters. As the Tokens and Context Windows post described, a tokenizer splits text into pieces from a fixed vocabulary. Common words become single tokens. A misspelled word becomes several rarer fragments, because the tokenizer has no entry for it.
Those fragments are not meaningless to the model. Web-scale training data is full of typos, so models have seen millions of misspellings next to their intended words, and they learn to reconstruct the intended word from the pieces and the surrounding sentence. A 2025 study went further and fed models deliberately unnatural token splits of ordinary text; post-trained models still read the meaning, perceiving the text as containing misspellings rather than as gibberish.
This repair is the reason casual typos usually go unnoticed. It also defines its own limits. Repair only works when something looks broken and the context points to a single plausible fix.
What the research measures
Robustness studies follow a common recipe. Take a benchmark with known answers, inject a controlled kind of noise into the questions, and measure how far accuracy falls. The numbers below come from different benchmarks and models, so compare the patterns rather than the exact figures across studies.
Typos
Typos hurt more than everyday experience suggests. A study presented at ICML 2026 added single-character typos (swapped letters, neighboring keys, and flipped capitalization) to 5% of the words in MMLU questions, leaving numbers untouched. Llama 3.1 8B fell from 66.7% to 51.0% accuracy. GPT-4o mini fell from 82.1% to 74.9%, and even the reasoning model GPT-5 nano dropped from 91.3% to 86.5%, although its reasoning traces often noted the typo.
The pattern holds across other work. A 2025 study of 18 open-weight models in 12 languages found that typos degraded every model and every task, with reasoning and generation tasks hit harder than classification. Larger models started higher but still lost a meaningful share of their accuracy, and instruction tuning sometimes made models more brittle to noise even as it improved clean performance.
Adversarial numbers are larger still. Microsoft's PromptBench chose perturbations specifically to break models and found character-level edits cut performance by about 20% on average, with word-level synonym swaps cutting about 33%. Treat those as worst cases. Real typos are random, not chosen by an attacker searching for the most damaging edit.
Grammar and informal writing
Grammar mistakes are the most forgiven kind of noise. A 2024 study that injected five noise types into MMLU found models handled grammatical errors better than OCR or speech errors, likely because grammatical mistakes are common in the text models are trained on. Across all five noise types, it measured accuracy drops between 2.5% and 8.2%.
The lesson is that broken grammar rarely changes meaning. "Me want refund for order not arrive" is ungrammatical but unambiguous. The model fills in structure it has seen countless times.
Mixed languages
Code-switching, moving between languages within a conversation or a sentence, is normal for billions of speakers and still costs accuracy. A 2026 benchmark covering 16 code-switched language pairs measured an average drop of about 11% on question answering, and found that mixing languages introduced unsupported claims and logical leaps into the models' reasoning.
Script matters too. In the multilingual typo study, languages with non-Latin scripts such as Arabic, Hindi, and Bengali saw much sharper drops under noise than Latin-script languages. The cost is not only accuracy: the same content can take several times as many tokens outside English, so noisy non-English input is both more expensive and more fragile.
OCR and speech noise
Errors produced by machines are harder than errors produced by people. The 2024 noise study found OCR and speech-recognition errors caused larger drops than typos or grammar, because these error patterns are rarer in training data. An OCR engine confuses rn with m and 1 with l; a transcriber turns "fifteen" into "fifty."
The ICML 2026 study gives a concrete number for speech. On Spoken-SQuAD, where passages are speech-recognition transcripts of the original SQuAD text, Llama 3.1 8B scored 49.5% exact match against 60.0% on the clean passages. Same questions, same facts, about ten points lost to transcription.
For retrieval systems, the damage compounds. OHRBench, a 2024 benchmark of OCR inside RAG pipelines, found that none of the OCR solutions it tested produced a knowledge base as good as the ground truth, with a gap of at least 7.5% even for the best one.
The benchmark also split OCR noise into two kinds. Semantic noise, wrong characters and words, consistently hurt both retrieval and generation. Formatting noise, such as tables and formulas rendered inconsistently, affected some retrievers and models far more than others. The Documents, Images, and Multimodal Input post covers how to parse documents to avoid this.
Formatting
Formatting is the noise that surprises people most, because it changes nothing a human would call meaning. A 2023 study varied only separators, casing, and spacing in few-shot prompts and found accuracy differences of up to 76 points on LLaMA-2-13B. The sensitivity persisted with larger models, more examples, and instruction tuning, and the best format for one model only weakly predicted the best format for another.
A 2024 study found the same effect at the template level. Rendering the same prompt as plain text, Markdown, JSON, or YAML changed GPT-3.5-turbo's performance by up to 40% on a code translation task, while GPT-4 was much more stable. Newer models are less sensitive than older ones, but no study has shown the effect disappearing, so measure it on your model rather than assuming.
Irrelevant content
Noise is not only corrupted characters. Extra, irrelevant information also costs accuracy. Google researchers added a single irrelevant sentence to grade-school math problems and found that, of the problems the models solved before, no more than 18% stayed solved across every kind of distraction. Adding an instruction to ignore irrelevant information helped, and so did sampling several answers and voting.
What actually needs to be clean
Put the studies together and a rule emerges. Models repair noise when the error looks like an error and the context points to one fix. They fail when the error looks valid, when nothing nearby disambiguates it, or when the same noise repeats in every call.
The things worth cleaning fall into a few groups:
- Words that flip meaning. "Do now delete" and "do not delete" are both fluent. Negations, "all" versus "any," "before" versus "after": a typo here produces a different valid instruction, not a broken one.
- Exact values. IDs, SKUs, account numbers, file paths, function and column names, prices, and dates have no near miss. The model will either copy your typo faithfully or "correct" it to something that does not exist.
- Anything a retriever matches. Keyword search misses a misspelled term outright, and research on dense retrievers has repeatedly measured significant drops on queries with typos. If retrieval fails, the model answers from the wrong context and never knows it.
- Machine-made text you control. OCR and transcription errors are harder for models than human typos, and they live in your knowledge base where every query touches them. Fix them once at ingestion rather than hoping the model repairs them a thousand times.
- Everything repeated on every call. System prompts, templates, few-shot examples, and tool descriptions. A typo in a user question affects one answer. A typo or inconsistent format in a template affects every answer, and models tend to imitate the style of their examples.
What usually does not need cleaning is the user's own conversational text. Spell-correcting chat messages before they reach a capable model rarely helps and can hurt, because correctors happily "fix" product names, code, and other languages.
How to test your own system
Published numbers tell you the shape of the risk. Only your own eval tells you the size. The cheapest robustness test is to take your existing eval set, add noise at a few rates, and see how far the score moves.
import random
import re
KEYBOARD = {"a": "qs", "e": "wr", "i": "uo", "o": "ip", "n": "bm", "t": "ry", "s": "ad", "r": "et"}
PROTECTED = re.compile(r"\d|[_/.:@-]|[A-Z]{2,}") # numbers, IDs, paths, acronyms
def add_typos(text: str, rate: float, seed: int = 0) -> str:
rng = random.Random(seed)
words = text.split(" ")
for i, w in enumerate(words):
if len(w) < 4 or PROTECTED.search(w) or rng.random() > rate:
continue
j = rng.randrange(1, len(w) - 2)
if rng.random() < 0.5: # swap two neighboring letters
w = w[:j] + w[j + 1] + w[j] + w[j + 2:]
elif w[j].lower() in KEYBOARD: # hit a neighboring key
w = w[:j] + rng.choice(KEYBOARD[w[j].lower()]) + w[j + 1:]
words[i] = w
return " ".join(words)
question = "Which plan includes priority support for invoice INV-4471 this month?"
for rate in (0.0, 0.2, 0.5):
print(rate, add_typos(question, rate, seed=7))The function introduces human-style typos into a chosen fraction of words: swapped letters and neighboring keys. It deliberately skips short words, numbers, IDs, and paths, so the test measures the noise a model is expected to repair. Run with seed 7, it turns "plan" into "paln" at 20% and produces "Wihch paln inclueds prioirty supprot" at 50%, while INV-4471 survives untouched.
Run your eval at 0%, 5%, and 10% noise with several seeds, and compare the scores. Then build a separate, small test set for the fragile cases from the previous section: flipped negations, wrong IDs, and OCR-style substitutions in retrieved text. Those should not be scored on whether the model recovers, but on whether your system notices, by validating IDs against a database or asking the user to confirm.
What goes wrong
Evals on clean data. The opening story is the most common failure. Test sets written by the team that built the product are grammatical, monolingual, and well formatted, so they measure the best case. Sample real traffic, with its mess, into the eval set.
Normalizing away meaning. Aggressive preprocessing does its own damage. Spell correction rewrites product names and code identifiers, lowercasing destroys case-sensitive keys, and translating everything to English loses nuance the model could have handled. Clean the parts you control and validate the parts that must be exact; leave conversational text alone.
Trusting the model to clean. A "re-pass" step, where one model call rewrites noisy input before another answers it, sounds appealing. The 2024 noise study tested exactly this and found that correcting noisy instructions was difficult, particularly for open-weight models, and only some models were reliable at it. A rewrite step can also silently change what the user asked.
Template drift. Two versions of a prompt template that differ only in separators or Markdown can score differently, and the difference may reverse on a new model. Keep templates under version control and rerun evals when either the template or the model changes. The Choosing a Model post covers how to build that eval.
Irrelevant context piling up. Long conversations, verbose tool results, and loosely matched retrieval chunks are noise too. The distraction research suggests trimming them helps accuracy as well as cost, which is the core argument of the Context Engineering post.
Cost, latency, and complexity
Most robustness fixes are cheap if you place them in the right stage. Validating IDs and numbers against your own data costs a database lookup. Fuzzy matching in keyword search, such as the fuzziness option in Elasticsearch and OpenSearch queries, adds little latency. Hybrid search, combining keyword and vector retrieval as described in the RAG post, covers more failure modes than either alone.
Ingestion-time cleanup costs once per document instead of once per query. Better document parsing or re-running OCR on low-confidence pages is almost always cheaper than accepting a permanently noisy knowledge base.
The expensive fixes are the ones that run on every request. An LLM re-pass adds a full model call of latency and cost, and sampling several answers to vote, which helped against distraction, multiplies output cost by the number of samples. Reserve these for high-stakes paths where the measured gain justifies them.
The current landscape
The research tools behind this post are open and usable for your own testing. PromptBench from Microsoft packages character-, word-, sentence-, and meaning-level perturbations. FormatSpread, from the 2023 formatting study, estimates the spread of accuracy across plausible prompt formats. OHRBench measures OCR's impact on RAG, and the 2025 MulTypo generator produces keyboard-aware typos for multiple languages.
Model robustness keeps improving with scale and training. In most studies, newer and larger models lose less, and reasoning models often notice and work around typos. But every study that tested frontier models still measured a drop, and the most stubborn sensitivities, formatting and machine-generated noise, have narrowed rather than disappeared. Assume tolerance, not immunity.
Summary
Models are robust to mess that looks like mess. A misspelled word or a broken sentence signals that something needs repair, and the surrounding context usually says what. That is why casual chat with typos works so well, and why it tells you little about your production system.
The failures come from noise that looks valid or that is everywhere at once: a typo that forms a different real word, a wrong ID, OCR errors baked into every retrieved chunk, an inconsistent template on every call. So spend cleaning effort where a mistake cannot be inferred from context, validate exact values instead of trusting them, and test your system with the input your users actually send.
Part of the Applied AI series: AI from the builder's side.