Structured Outputs: How to Get Reliable JSON From an LLM

There are three levels of enforcement for getting JSON out of a model, and they are not interchangeable. Asking politely in the prompt gives you no guarantee. JSON mode guarantees the output parses, but not that it has the fields you asked for. Schema enforced structured output guarantees the shape, because the model is physically prevented from emitting a token that would break it. Knowing which one you are using determines whether you still need validation code, and the answer is that you always do.

This guide covers how each level works, the accuracy tradeoff nobody mentions, and how to design schemas that do not fight the model.

The three levels of enforcement

Level one: ask in the prompt. You write “respond only with JSON” and hope. The model usually complies and sometimes wraps the output in explanatory text, adds a markdown code fence, or trails off. This is best effort with no guarantee at all, and it is where most parse failures come from.

Level two: JSON mode. A parameter that constrains output to syntactically valid JSON. The braces will balance and the quotes will close. It says nothing about whether the object contains the fields you need, whether types are right, or whether a required value is present. Valid JSON and correct data are different claims.

Level three: schema enforced structured output. You supply a schema, and the provider enforces it during generation. Every field you defined appears, with the type you defined, and nothing extra. This is the only level that gives you a structural guarantee.

The distinction between levels two and three is the one that costs people time. A team switches on JSON mode, sees clean parsing in testing, and ships. Weeks later something downstream breaks because a field the code assumed was present came back missing. Nothing errored, because a JSON object without your field is still valid JSON.

How enforcement actually works

Understanding the mechanism explains both the guarantee and its limits.

A model generates text one token at a time. At each step it produces a score for every token in its vocabulary, tens of thousands of candidates, then samples one.

Constrained decoding inserts a filter into that loop. Your schema is compiled into a state machine that knows, at every position, which tokens could still lead to a valid result. Before sampling, the filter masks out every token that would violate the schema, effectively setting their probability to zero. The model cannot choose them, because they are no longer available to choose.

That is why the guarantee is structural rather than probabilistic. The model is not trying hard to follow your schema and mostly succeeding. It is prevented from deviating.

Two consequences follow.

Compiling the schema has a cost. Turning a schema into a state machine takes work, which matters for high throughput pipelines. Serving engines cache compiled grammars so the cost is paid once rather than per request.

A guarantee about shape is not a guarantee about content. The model will return a field called date containing a string matching your pattern. Whether that string is the right date is entirely unconstrained. Enforcement operates on structure, never on truth.

The tradeoff nobody mentions

Here is the part most guides skip, and it is worth knowing before you enforce schemas everywhere.

Research on format restriction has found that constraining output can reduce task accuracy. A benchmark comparing plain JSON generation against constrained decoding found plain generation achieved better accuracy overall, with constrained decoding’s clearest advantage being lower token usage, and with meaningful accuracy degradation on some models.

The intuition is reasonable once you see it. Constraining the token space is a real intervention in how the model generates. Forced into a rigid structure from the first token, the model has less room to work through the problem in the way it otherwise would.

This does not mean you should abandon schema enforcement. For high volume extraction where the task is mechanical and reliability matters more than marginal quality, the guarantee is worth it, and it is exactly the workload where structured output earns its place. But treat it as a tradeoff rather than a free upgrade, and if a task is genuinely hard, test both ways rather than assuming enforcement is strictly better.

Let it think first, then constrain

There is a practical resolution to that tradeoff, and it is the most useful technique in this guide.

The problem with constraining from the first token is that reasoning and formatting get forced into the same pass. The fix is to separate them: let the model reason in free text, then produce the structured result.

The simplest version puts a reasoning field first in your schema, before the answer fields. Since generation is sequential, the model works through the problem in that field and then fills in the conclusions, and JSON key order in your schema is the order it generates in. You pay for the reasoning tokens, and on a hard task it usually buys back the accuracy that constraint costs.

The heavier version splits it into two calls: an unconstrained pass to reason, then a constrained pass to format the result. More expensive and more reliable, and worth it when correctness matters more than cost.

For mechanical extraction, skip both. There is nothing to reason about, and the reasoning field is pure overhead.

Designing schemas that do not fight the model

Keep it flat where you can. Deeply nested structures are harder for models to populate correctly and produce more scaffolding tokens. If the task allows a flat object, use one.

Name fields the way a human would. Field names are tokens the model reads. invoice_total communicates more than f3, and the model uses that meaning to decide what to put there.

Prefer enums over free strings for categories. If a field should be one of five values, define it as exactly those five. Enforcement then makes an invalid category impossible instead of merely unlikely, which removes a whole class of downstream mapping code.

Handle optional fields with placeholders, not complex conditionals. Rather than expressing intricate optionality in the schema, include the fields and let unneeded ones take a null or empty placeholder, then strip them afterwards. This is simpler to express and cheap under constrained decoding, because the scaffolding is largely not generated as free tokens.

Split large schemas. A schema with fifty fields asks the model to hold a lot in mind at once. Two calls with focused schemas often produce better data than one call trying to extract everything.

Watch for interactions with tool use. An active output grammar restricts what tokens can be emitted at all, which can conflict with a model’s ability to emit tool calls in the same turn. If you are combining schema enforcement with tool use, test that combination specifically rather than assuming they compose.

Failures still happen, and they cost money

Even at level three, you need error handling, because a structural guarantee does not cover everything that can go wrong.

Truncation. If the response hits your token limit mid object, you get incomplete output. The schema was being followed right up until generation stopped. Size max_tokens for your largest realistic response, and remember that on models which reason before answering, reasoning tokens consume the same budget.

Everything that is not generation. Rate limits, timeouts, and context overflows are unaffected by output constraints. They fail the way they always did.

Semantically wrong data in a perfectly valid shape. The most dangerous failure, because nothing catches it. Your parser is happy, your types check, and the value is wrong.

This is also where structured output connects to your bill. Every failed request is billed in full. A pipeline that retries on parse failures is paying for each attempt, so a ten percent failure rate is a ten percent cost increase on top of the latency and complexity. That is the real argument for level three on high volume work: not elegance, but the retries you stop paying for.

So validate anyway. Parse into a typed model, check the values make sense for your domain, and log the raw response when something fails so you can see what actually came back rather than guessing.

When you route across providers

Two things matter if your workload can land on more than one model.

Enforcement strictness varies. Some providers genuinely guarantee schema conformance through constrained decoding. Others accept a similar looking parameter and treat the schema as a strong hint, which produces conforming output most of the time. Code written against a guarantee will break against a hint, and it will break intermittently, which is the worst way to find out.

Capability is not uniform. Support for the more advanced parts of JSON Schema differs between implementations. A schema that works on one may be rejected or partially honored on another.

The practical approach is to design to the weakest guarantee you might actually run against. Keep schemas conservative, keep validation in place regardless of what the provider promises, and test your real schema against each model you might route to rather than trusting the feature name.

MixRoute puts every major model behind one OpenAI compatible endpoint with zero markup, which makes testing a schema across models a loop over model strings rather than several integrations built to run the comparison. Start building on MixRoute

FAQ

How do I get an LLM to return valid JSON? There are three levels. Prompting alone gives no guarantee. JSON mode guarantees syntactically valid JSON but not that it contains the fields you asked for. Schema enforced structured output guarantees the shape, because invalid tokens are blocked during generation. Use the highest level your provider supports, and validate the result regardless.

What is the difference between JSON mode and structured outputs? JSON mode guarantees the output parses as JSON. Structured outputs guarantee the output matches a schema you define, including which fields exist and what types they hold. JSON mode will happily return a valid object missing the field your code depends on, which is why the distinction matters in production.

How does constrained decoding work? Your schema is compiled into a state machine that tracks which tokens could still produce a valid result. At each generation step, tokens that would violate the schema are masked out before sampling, so the model cannot select them. The guarantee is structural rather than probabilistic, because deviating options are removed rather than discouraged.

Does structured output reduce answer quality? It can. Research on format restriction has found that constraining generation can lower task accuracy compared with unconstrained output, with the effect varying by model and task. For mechanical extraction the guarantee is usually worth it. For harder reasoning tasks, let the model reason in a free text field first and constrain only the final answer, or test both approaches.

Do I still need validation if the schema is enforced? Yes. Enforcement guarantees shape, never correctness. A response can match your schema perfectly and contain wrong values. Responses can also truncate mid object if they hit the token limit, and rate limits, timeouts, and context overflows are unaffected by output constraints. Parse into a typed model and validate the values.

Why do my structured output calls sometimes fail? The most common causes are truncation from a token limit set too low, schema features the provider does not fully support, and interactions with other features such as tool use. Log the raw response on failure rather than only the parse error, since the raw output usually shows immediately which of these occurred.

The bottom line

Getting reliable JSON is a question of which guarantee you are actually holding. Prompting is a hope, JSON mode is a syntax promise, and schema enforcement is a structural one delivered by blocking invalid tokens during generation.

Enforcement is the right default for high volume extraction, where it removes a class of parse failures you would otherwise be paying to retry. It is not free: constraining generation can cost accuracy, so on harder tasks let the model reason in an unconstrained field first and constrain only the answer.

And validate regardless of what the provider promises. A guarantee about shape is never a guarantee about truth.

MixRoute gives you every major model behind one OpenAI compatible endpoint with zero markup, so testing how a schema behaves across models is a string change rather than a project. Start building on MixRoute