Information in this post reflects publicly available sources as of October 17, 2026.
Your team builds a pipeline that reads incoming support tickets and asks a model to classify each one: a category, a priority, the customer's email, and a one-line summary. The prompt ends with "Respond in JSON." Your code calls json.loads on the reply and routes the ticket. In testing, it works every time.
In production, it works almost every time. One reply in a few hundred begins with "Sure! Here's the classification:" before the JSON. Another wraps the JSON in a Markdown code fence. One has a trailing comma, and one sets the priority to "urgent-ish", which your routing code has never heard of.
The pipeline does not crash every day, which is worse: it crashes at 2 a.m. on a Saturday, or it quietly drops tickets.
Asking a model for JSON is a request, not a guarantee. Structured outputs turn it into a guarantee, at least for the shape of the data. This post covers how that works, how to design schemas that models fill in well, and the part that no API can guarantee for you.
Text in, text out, but code needs structure
A language model produces text one token at a time. Nothing about that process knows or cares that your code expects an object with four fields. A good prompt makes valid JSON very likely, and modern models follow format instructions well. But "very likely" across a million calls still means thousands of failures.
This matters far beyond ticket classification. Every tool call an agent makes is structured output: a tool name plus arguments that must match a schema. Every extraction job, every classifier, every router that decides which agent handles a request depends on the model producing data that code can consume. The more of your system runs on model output, the more those rare failures add up.
How it works
Four levels of guarantee
There are several ways to ask for structure, and they promise very different things.
With prompt-only formatting, you describe the format in the instructions and hope. JSON mode, offered by several providers, guarantees syntactically valid JSON but says nothing about which keys appear or what types they hold. Schema-constrained output, which providers call structured outputs or strict mode, guarantees that the result matches a JSON Schema you supply: every required field present, every enum value one you listed, no extra keys.
The top step is validation in your own code. A schema can say a field is a string containing an email address. It cannot say whether that address belongs to the customer who wrote the ticket.
Constrained decoding
Schema-constrained output is not a better prompt; it changes how the model picks tokens. At each step, the model produces a probability for every possible next token. Normally the system samples from that distribution. With constrained decoding, the provider first compiles your schema into a grammar, and at every step it masks out any token that would make the output invalid under that grammar.
That is how the guarantee works. The model's favorite next token, urgent, is not one of the allowed values, so it is never sampled. The preamble Sure is impossible too. Whatever the model would have preferred, every token it actually emits keeps the output inside the schema.
The example also shows the subtle cost. The model wanted to say urgent, and the grammar forced it to pick from what remained, and it picked high. That is probably fine here, but it is a reminder that the constraint shapes the content as well as the format. If your enum is missing an option the model needs, it will be pushed into the nearest wrong one without any sign that anything happened.
Structured output also streams. The tokens arrive one at a time like any other response, and partial JSON parsers in most SDKs can surface each field as soon as it is complete. A user interface can show the category and priority while the summary is still being written, instead of waiting for the closing brace.
Tool calling uses exactly the same machinery. A tool's input schema is a structured output schema, and providers now offer a strict mode for tool definitions that guarantees the arguments match. When the MCP post described a tool's inputSchema, this is what enforces it.
In code
In practice, you rarely write JSON Schema by hand. You define a type in your language, usually with Pydantic in Python or Zod in TypeScript, and the SDK converts it into a schema and parses the result back into an object:
from typing import Literal
from pydantic import BaseModel, Field
from openai import OpenAI
class Ticket(BaseModel):
reasoning: str = Field(description="One or two sentences on why you chose the category and priority.")
category: Literal["billing", "bug", "account", "feature_request", "other"]
priority: Literal["low", "medium", "high"]
customer_email: str | None = Field(description="Email address from the ticket, or null if none is given.")
summary: str = Field(description="One sentence, under 20 words.")
client = OpenAI()
completion = client.chat.completions.parse(
model=MODEL,
messages=[
{"role": "system", "content": "Classify the support ticket."},
{"role": "user", "content": ticket_text},
],
response_format=Ticket,
)
message = completion.choices[0].message
if message.refusal:
send_to_review_queue(ticket_text, reason=message.refusal)
else:
ticket = message.parsed # a Ticket object, already validated against the schemaThis uses OpenAI's Python SDK, but the shape is the same on other providers: Anthropic's API takes a JSON Schema through its output_config parameter, and Google's Gemini API through response_json_schema. In every case, the type definition is the contract, and the result arrives as a typed object instead of a string to parse.
Designing schemas that models fill in well
Notice a few choices in that schema, because each one prevents a common failure.
Reasoning comes first. Models generate fields in order, so a field that comes before the answer gives the model room to think before committing. A 2024 study, "Let Me Speak Freely?", found that strict JSON formats could sharply reduce accuracy on reasoning tasks, partly because models put the answer key before the reasoning. Putting reasoning first, or answering in plain text and converting to JSON in a second step, avoids most of that loss.
Enums instead of free text. Literal["low", "medium", "high"] makes "urgent-ish" impossible. Every field your code branches on should be an enum, and the list should include an escape value such as "other" so the model is never forced into a wrong category.
Null instead of invention. If a required field has no honest value, a model under a strict schema will make one up. Making customer_email nullable, and saying so in its description, gives the model a legitimate way to report that the email is not there.
Descriptions are instructions. Field descriptions travel with the schema to the model. "One sentence, under 20 words" in a description works better than the same rule buried in a long system prompt.
Valid is not the same as correct
Constrained decoding solves syntax and shape. It cannot solve meaning. A schema-valid ticket can still have the wrong category, an email address the model invented, or a summary that contradicts the ticket. That is why the top step of the staircase belongs to your code.
Some checks are mechanical: is the extracted email actually present in the ticket text, is the date in the future, does the product ID exist in your catalog? Pydantic validators or ordinary code handle these well. When a check fails, retry with the specific error message, such as "the email you returned does not appear in the ticket," which gives the model something concrete to fix. Cap the retries and send persistent failures to a person.
Accuracy across the whole pipeline is an evals question. A schema makes every output parseable, which is exactly what makes it easy to score hundreds of them automatically against known answers.
When to use it, and when not to
Use schema-constrained output whenever code, rather than a person, consumes the result: extraction, classification, routing, tool arguments, and any step where a downstream system branches on a field. Once your provider supports strict mode for your model, there is little reason to rely on prompt-only JSON for these jobs.
Constraints can even improve accuracy on some tasks. The same study that found reasoning losses under strict formats found gains on classification: limiting the answer to a fixed set of labels stopped models from drifting into near-miss answers. Routing and labeling jobs are where schema constraints help most.
Do not use it for text meant for people. Forcing an answer to a user into a JSON field adds nothing and can make the writing stiffer. Be careful with tasks that need extended reasoning: either give the schema a reasoning field that comes first, or let the model think in plain text and convert the conclusion in a separate step.
Also check what your provider's schema support actually covers. Providers implement different subsets of JSON Schema. A constraint such as a numeric minimum or a string length may be unsupported, and some SDKs quietly move unsupported constraints into the field description rather than enforcing them. Anything the grammar does not enforce, your validation must.
What goes wrong
Required fields get invented. A strict schema with a required order_id will get an order ID, whether or not the input contains one. Make fields nullable when the information may be missing, and check extracted values against the source text.
Truncation and refusals break the guarantee. If the response hits the token limit mid-object, the output is cut off and will not parse. If the model refuses a request, providers return a refusal instead of schema-valid data. Always check the stop reason or refusal field before trusting the payload, and set token limits with room to spare.
Missing enum values force wrong answers. As the masking example showed, the model will always pick something from your list. If tickets arrive in a category your schema does not include, they will be silently misfiled. Include an "other" value and monitor how often it appears.
Reasoning quality drops. In the "Let Me Speak Freely?" study, one model's accuracy on grade-school math problems fell from about 77% in plain text to about 49% under strict JSON mode. Newer models are less sensitive, but the remedy is cheap: put a reasoning field first, or split reasoning and formatting into two steps.
Schema changes break consumers. Renaming a field or removing an enum value is an API change for every system downstream. Version your schemas like any other contract, and run your evals before changing them.
Cost and latency
Structured outputs are usually cheaper than free text, because JSON answers are short and skip the pleasantries. Take a batch of 10,000 tickets, each with about 600 input tokens and 120 output tokens. At an illustrative $3 per million input tokens and $15 per million output tokens, the batch costs about $36.
The schema itself adds input tokens, since providers describe the format to the model behind the scenes, and long field names repeat in every object of a large list. The first request with a new schema can also be slower while the provider compiles its grammar. Anthropic, for example, caches compiled grammars for 24 hours after last use, so the delay is paid once per schema rather than once per call.
Retries cost little in money and a lot in latency. If 2% of tickets fail your business checks and need one retry, the batch costs about $0.72 more, but each affected ticket waits for a second full model call. The more expensive failure is the one you never see: schema-valid data that is quietly wrong.
The current landscape
All three major API providers now offer schema-constrained output. OpenAI introduced Structured Outputs in August 2024, with a strict mode for both response formats and function definitions. Anthropic added structured outputs in November 2025, covering JSON responses through output_config.format and strict tool use, and no longer requires a beta header. Google expanded the Gemini API's JSON Schema support in November 2025, adding keywords such as anyOf and $ref and returning fields in schema order.
The supported subsets differ. Anthropic's documentation, for example, lists numeric bounds and string length limits as unsupported, while Gemini added numeric minimum and maximum. Check the provider's schema documentation before relying on any constraint beyond types, required fields, and enums.
For open-weight models, inference servers such as vLLM and SGLang provide structured output through grammar engines including XGrammar, llguidance, and Outlines. Libraries such as Instructor add validation and automatic retries on top of any provider, and BAML takes a schema-first approach with its own definition language. On the protocol side, MCP tools declare JSON Schemas for their inputs and, optionally, their outputs.
Summary
Structured outputs turn the shape of model output from a probability into a guarantee. Constrained decoding does this by removing invalid tokens before the model can choose them, which is why it works on every call instead of most of them. For any output that code consumes, it should be the default.
But a guaranteed shape is not a guaranteed truth. The schema decides what the model is allowed to say, so design it with care: reasoning before answers, enums with an escape value, nullable fields for missing information. Then validate the values yourself, because the staircase always ends with your code.
Part of the Applied AI series: AI from the builder's side.