Information in this post reflects publicly available sources as of November 1, 2026.
An accounts payable team automates invoice processing with a vision model. It reads scanned invoices, extracts the vendor, date, and total, and posts them to the ledger. For three weeks it works beautifully. Then an auditor finds a $7,340 invoice paid as $1,340: a smudged 7 on a faxed scan, read confidently as a 1, and written into perfectly valid JSON.
Down the hall, the same company's document search answers a question about quarterly revenue from a PDF report. The answer is wrong because the text extractor read a two-column page straight across, splicing sentences from both columns, and skipped the chart that held the actual figure.
Most of the information businesses care about is not clean text. It lives in PDFs, scans, slides, screenshots, forms, charts, and photos. Modern models can read all of these, but how they read them, and how they fail, is different from text. This post covers working with documents and images as a builder.
The problem: the world is not plain text
Early language model applications assumed text in and text out. Teams converted everything to text first, and the conversion silently discarded whatever did not survive it: the layout that shows which number belongs to which label, the table structure, the chart, the handwritten note in the margin, the checkbox that is ticked.
Multimodal models remove that bottleneck by reading images directly. All the major model families now accept images, and most accept PDFs. That makes new things possible, from reading a screenshot of an error to understanding a chart, but it also brings new costs, new limits, and new ways to be confidently wrong.
How it works
How a model sees an image
A vision model does not look at an image the way a person does. It cuts the image into small square patches, turns each patch into a token, and reads those image tokens in the same sequence as the text tokens of your question.
That has two practical consequences. First, images cost tokens, and the cost grows with resolution.
With Anthropic's models, for example, each 28-by-28-pixel patch is one token, so a 1000-by-1000-pixel image costs about 1,300 tokens. OpenAI uses patches or tiles depending on the model, and Google's Gemini counts 258 tokens per tile. A long PDF sent as page images can fill a context window quickly.
Second, there is a maximum resolution. Images larger than the limit are downscaled before the model sees them, and detail that does not survive downscaling is gone. Small print, dense spreadsheets, and fine lines on a chart are the first casualties. Crop to the region that matters, or split a large page into pieces, rather than sending a huge image and hoping.
PDFs are two different things
A digital PDF, exported from a word processor or a reporting tool, contains a text layer: the characters are stored as text, and a library can extract them exactly. A scanned PDF is just pictures of pages. Extracting text from it requires optical character recognition (OCR), which can misread characters, and some scans carry a hidden OCR text layer that may be wrong in ways nobody can see.
Even a perfect text layer loses structure. Text extraction may read multi-column pages in the wrong order, flatten tables into a stream of numbers, and drop charts and images entirely. Native PDF support in model APIs addresses part of this by sending the model both the extracted text and an image of each page, so it can use the layout as well as the characters. That is more accurate and also more expensive, since every page now costs image tokens too.
Three pipelines
Applications that work with documents tend to use one of three designs, and the choice drives cost, accuracy, and what you can search.
Parse first converts documents once, with a layout-aware parser, into Markdown or structured JSON that keeps headings, reading order, and tables. Every later question reads cheap text, and the text can be indexed for search. The weakness is anything the parser cannot capture, especially charts and complex visual layouts.
Send the pages to a vision model on every question. The model sees exactly what a person would, including charts and layout. It costs more per question, and it does not scale to searching thousands of documents, because you cannot send them all.
The hybrid approach combines them: parse and index everything for search, then send the images of the few pages that search finds to a vision model for the answer. This "visual RAG" pattern, an extension of the RAG post's design, gets the economy of text search and the accuracy of reading the actual page.
Extracting structured data
One of the most valuable uses of vision models is turning documents into data: invoices into ledger entries, forms into records, receipts into expense reports. Combine image input with structured outputs so the result always matches your schema, and then check it, because a valid schema says nothing about whether the values are right.
import base64, json
import anthropic
client = anthropic.Anthropic()
schema = {
"type": "object",
"properties": {
"invoice_number": {"type": "string"},
"invoice_date": {"type": "string", "description": "ISO 8601 date"},
"line_items": {"type": "array", "items": {
"type": "object",
"properties": {"description": {"type": "string"}, "amount": {"type": "number"}},
"required": ["description", "amount"], "additionalProperties": False}},
"total": {"type": "number"},
},
"required": ["invoice_number", "invoice_date", "line_items", "total"],
"additionalProperties": False,
}
page = base64.b64encode(open("invoice_page1.png", "rb").read()).decode()
resp = client.messages.create(
model="claude-sonnet-5-5",
max_tokens=2048,
output_config={"format": {"type": "json_schema", "schema": schema}},
messages=[{"role": "user", "content": [
{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": page}},
{"type": "text", "text": "Extract the invoice fields exactly as printed."},
]}],
)
invoice = json.loads(resp.content[0].text)
items_sum = round(sum(item["amount"] for item in invoice["line_items"]), 2)
if abs(items_sum - invoice["total"]) > 0.01:
print(f"Needs review: items sum to {items_sum}, total reads {invoice['total']}")The image goes before the text, as Anthropic's documentation recommends, and the schema guarantees parseable output. The last lines are the important part: they compare the line items with the total. A misread digit in either place breaks the arithmetic, and the invoice goes to a person instead of the ledger.
Arithmetic is one check among several. Validate formats such as dates and tax IDs, and compare against records you already hold, such as the purchase order, the vendor list, or last month's invoice from the same supplier. Route anything that fails to a person who sees the page image beside the extracted fields, so review takes seconds rather than minutes.
Screenshots and interfaces
Screenshots are a special case worth knowing. A screenshot of an error dialog, a dashboard, or a misbehaving web page often carries more context than a description of it, and models read interface text well because it is crisp and high-contrast. Support tools increasingly let users attach screenshots for exactly this reason.
The same ability powers agents that operate software by looking at the screen, covered in the computer use post. Those agents take a screenshot after every action, which makes image token costs and resolution limits a central design concern rather than a detail.
Audio and video, briefly
Audio and video follow the same principle: the input becomes tokens, and the token count drives cost and latency. Google's Gemini models accept video directly, sampling frames and counting tokens for each second of footage, so an hour of video is a very large input. For most applications, a cheaper pipeline works better: transcribe the audio, sample a few key frames, and send the model the transcript and the frames that matter.
Speech in conversation is a different problem, dominated by latency rather than understanding, and the voice agents post covers it.
When to use it, and when not
Vision input shines where meaning lives in the layout or the pixels: charts and diagrams, forms with checkboxes, screenshots of applications and error messages, slides, photos of whiteboards, receipts, and scanned paperwork. It is also often the simplest way to handle messy real-world documents that defeat rule-based parsers.
It is weaker at tasks that need pixel precision or exhaustive counting. Models give approximate positions and approximate counts, especially for many small objects. They misread small or degraded text, cannot reliably tell whether an image is AI-generated, and are not substitutes for specialist systems in areas such as medical imaging.
And when clean text exists, use it. If a document has a reliable text layer, or the data exists in a database or an API, sending a picture of it is slower, costlier, and less accurate. Vision is for the information that only exists visually.
What goes wrong
Confident misreads. A smudged 7 becomes a 1, an O becomes a 0, a decimal point disappears. The output looks just as certain as a correct read. Validate numbers against arithmetic and other records, and send low-quality scans to review.
Downscaled detail. A full-page screenshot of a dense spreadsheet is shrunk before the model sees it, and the small cells become guesses. Crop, split, or increase resolution for the parts that matter, within the provider's limits.
Broken tables and reading order. Parsers merge cells, shift columns, or read across two columns at once. Test parsers on your own documents, especially the ugliest ones, and keep page references so answers can be checked against the original.
Charts read as estimates. A model reading a bar chart estimates values from pixels, so the numbers are approximate. When exact figures matter, find the underlying data instead of the picture of it.
Personal data in pixels. Scans and screenshots carry names, addresses, signatures, and account numbers that text-based redaction never sees. Apply the same data rules to images as to text, and crop or mask sensitive regions before sending, as the privacy post described.
Instructions hidden in images. Text in an image, including text too faint for a person to notice, is read by the model like any other input. A document or web page image can carry a prompt injection, so treat image content as untrusted, as the prompt injection post described.
Cost, latency, and complexity
Image tokens add up. At roughly 1,000 to 1,500 tokens per page image, a 300-page document sent as images approaches half a million tokens, which exceeds many context windows and costs real money on every question. Sending only relevant pages, at the lowest resolution that keeps the text legible, is the main lever. Prompt caching helps when the same document is asked about repeatedly.
Parsing shifts the cost to ingestion. Managed parsers typically charge from about a dollar to tens of dollars per thousand pages, depending on how much layout and table understanding you need, and open-source parsers cost only the compute to run them. Parse once, and every later question reads text at text prices.
Latency follows tokens. A question about one page image answers in seconds; a question that sends fifty pages takes noticeably longer to process before the first word appears. For interactive use, retrieve first and send few pages.
The current landscape
All the major model families from OpenAI, Anthropic, and Google accept images, and their APIs accept PDFs directly, combining extracted text with page images. Resolution limits and token formulas differ by provider and change between model generations; Anthropic's newest models, for example, accept higher-resolution images than earlier ones. Many open-weight models are also multimodal. Google's Gemini models additionally accept video and audio, counted in tokens per second.
For parsing, open-source options include IBM's Docling, released under the MIT license, and the Unstructured library. Hosted services include LlamaParse, Azure AI Document Intelligence, Google Cloud Document AI, Amazon Textract, and Mistral's OCR models. Rankings on public benchmarks such as OmniDocBench shift often, and the most widely used benchmarks are approaching saturation, so test candidates on a sample of your own documents.
Summary
Multimodal models let applications read the world as it actually arrives: scanned, photographed, laid out, and charted. But a model reading an image is still guessing at pixels, and its guesses look exactly as confident as its certainties. The skill is not in sending images to a model; it is in deciding what to send, at what resolution, and how to check what comes back.
So parse what you will search, send pages where layout or charts carry the meaning, crop to what matters, and validate every extracted number against something you already trust. Use text whenever text exists. And keep a person in the loop for the documents where a misread digit costs real money.
Part of the Applied AI series: AI from the builder's side.