指南
What is Structured Output / JSON Mode?
How structured output works, when to choose it over tool calling, and which models enforce JSON correctly.
Structured output (also called JSON mode or response_format=json_schema) constrains an LLM to emit a JSON document that conforms to a schema you provide. Unlike a prompt that asks "reply in JSON", structured output is enforced at decode time — the model literally cannot emit invalid JSON or miss a required field. This is the most reliable way to pipe LLM output into a typed downstream system.
How it's enforced under the hood
Strict JSON mode works by constrained decoding. At each step the model proposes a probability distribution over its full vocabulary, then the runtime masks out any token that would produce JSON not matching the supplied schema. The result is a single sample that is guaranteed to validate.
Looser "JSON mode" (no schema) only guarantees the output parses as JSON — fields can still be missing or have unexpected types.
Tool calling vs structured output
These features overlap but optimize for different cases:
- Structured output — single fixed schema, deterministic shape. Best for extraction, classification, summarization-with-fields.
- Tool calling — multiple possible actions; the model decides which to invoke. Best for agents, search-augmented QA, multi-step workflows.
- Real systems often combine both: a tool whose argument schema is itself a structured-output JSON schema.
Common pitfalls
Even with strict JSON mode, two failure modes remain:
- Schema-too-loose — a string field accepts any string. Constrain enums, regex patterns and length where possible.
- Empty or stub answers — the model technically returns valid JSON but every field is null. Add explicit "if you don't know, answer X" guidance and set realistic temperature.
Frequently asked questions
How is this different from prompting 'reply in JSON'?
Structured output is enforced at decode time — the model is constrained to emit tokens that produce valid JSON matching your schema. Plain 'reply in JSON' prompts can still produce malformed output, especially under load or adversarial input.
Can I use structured output with tool calling?
Yes, and it's a common pattern. Tool calling already enforces structured arguments; structured output adds the same guarantee to the model's final answer.
Are there latency or quality trade-offs?
Latency is usually within 5% of unconstrained generation. Quality can drop slightly for very tight schemas — the model has fewer 'escape hatches' to recover from a bad start.