Structured Outputs: Why Production AI Needs Schemas, Not Just Prose

The easiest AI demo ends with text. Production systems rarely do. They need a ticket object, a routing decision, a list of entities, an API payload, a tool call, or UI state that another program can consume without guessing what the model meant.

That is why structured outputs are more than a formatting trick. They are one boundary where a probabilistic model meets deterministic software. JSON mode can make syntax predictable. A schema can constrain shape. Pydantic or equivalent application validation can enforce types and local invariants. Business validators check facts and domain rules the schema cannot know. Authorization decides whether a structurally valid object may cause an external action.

1. Four Levels of Structure

free-form text
   ↓
valid JSON
   ↓
schema-conforming object
   ↓
business-valid object
   ↓
authorized action

Each level addresses a different failure mode. Free-form text is expressive but expensive for software to interpret reliably. JSON mode makes parsing predictable. Strict structured-output paths can constrain an object to a declared schema. Application validation checks rules the schema either cannot express or should not be trusted to decide alone—credit limits, resource ownership, deployment environment, whether a customer ID exists, or whether a timestamp is fresh enough for the workflow. Authorization is separate again: a valid request can still be forbidden.

JSON mode

Use JSON mode when you need machine-readable JSON but the exact shape is flexible. TrueFoundry's current Chat Completions documentation describes json_object as valid JSON without structure constraints. That is useful for lightweight extraction, but it should not be confused with a stable API contract.

JSON Schema

Use a strict schema when downstream software depends on a stable machine contract and the selected model/provider path supports the schema you need. Required fields, enums, nested structures, arrays, and additionalProperties rules can make the interface substantially more testable.

But “JSON Schema support” is not one universal capability. OpenAI's native Structured Outputs, for example, can constrain successful, non-refused, non-truncated responses to a supplied schema. Other providers support different subsets or semantics. TrueFoundry can bridge unsupported native paths by converting a response schema into a required tool call, but its own provider guidance documents provider-specific constraints—for example, some Anthropic paths do not accept numeric/string constraint keywords such as minimum/maximum or Pydantic's ge/le. A portable API surface is therefore not the same thing as a perfectly portable schema vocabulary.

Semantic and business validation

Even perfect structural adherence can be wrong. A model can emit a syntactically valid date that falls outside the permitted booking window, an order ID with the right type but the wrong tenant, or a refund amount that exceeds policy. Deterministic validators should handle rules that can be expressed deterministically before another model is asked to judge them.

Also distinguish provider conformance from application validation. If a provider or gateway returns a schema-conforming object, your application may still need Pydantic, Zod, a JSON Schema validator, database lookups, policy checks, and postconditions before trusting the object.

Action contracts

Tool calls take the same typed-contract idea into a higher-consequence setting: the object selects an operation and supplies arguments. Schema design therefore becomes part of security design. Narrow enums, explicit required fields, bounded objects, and minimal argument surfaces reduce what the model is able to propose—but authorization and policy must still decide what the caller is actually allowed to do.

Schema design is model-interface design

The best model-facing schema is rarely the schema your backend already exposes. Model interfaces benefit from small enums, explicit required fields, shallow nesting, and names that reflect the decision the model is actually making. If the backend accepts a 60-field object, consider asking the model for a five-field intent object and letting deterministic application code expand it into the internal representation.

This reduces prompt complexity and blast radius. A model that can propose {"action":"refund","order_id":"...","amount":...} has a narrower authority surface than one asked to manufacture an entire billing-service payload. Typed interfaces are therefore not only a reliability technique; they are a way to limit the state the model is permitted to invent.

Version schemas like APIs

Once downstream code depends on model output, schema changes are API changes. Adding a required field, renaming an enum, or changing nesting can break consumers even when the model itself is behaving correctly. Treat schema versions as deployable artifacts: record the version alongside requests or traces, test old and new variants against representative traffic, and make compatibility policy explicit.

A useful rollout pattern is shadow validation. Keep production generation on the current contract while validating representative outputs against a candidate schema in audit or offline evaluation. That reveals compatibility failures before the new contract becomes production-critical.

Constrained generation and repair solve different problems

When a model/provider supports native schema-constrained generation, use it: preventing structurally invalid tokens is better than parsing malformed output after the fact. But keep an explicit application failure path because refusals, interrupted generations, provider-specific schema limits, and business-validation failures still exist even when structural conformance is strong.

When repair is appropriate, make the defect machine-readable and keep the loop bounded. A small number of targeted corrections is usually easier to reason about than an open-ended “try again” cycle. For deterministic defects, prefer deterministic correction or escalation over asking a model to rediscover the same rule repeatedly.

2. Where TrueFoundry Fits: One Response Contract Across Models, With Provider-Specific Semantics

TrueFoundry's structured-output API supports JSON mode, JSON Schema, and Pydantic-style integration through the OpenAI-compatible Chat Completions surface. Its provider-support documentation makes the portability mechanism explicit: the Gateway uses a provider's native structured-output capability where available and, for other paths, can convert the response schema into a required tool call and extract the resulting arguments back into message.content.

That is useful portability, but it should not be read as identical semantics across every backend. Provider-native schema subsets differ, and a schema translated through tool calling is a different enforcement mechanism from native constrained decoding. Teams should test the exact model/provider combination they intend to ship and keep application validation at the boundary.

For Python applications, TrueFoundry also documents direct Pydantic validation and an Instructor integration. Instructor can turn Pydantic validation errors into retry feedback, but that retry behavior belongs to the Instructor/application workflow—not to “JSON Schema” as a concept.

At the action boundary, MCP Gateway carries typed tool contracts into governed external actions. The MCP 2026-07-28 specification defines both inputSchema and optional outputSchema; both default to JSON Schema 2020-12 when no $schema is supplied. MCP servers must return structured results conforming to a declared output schema, while clients are expected to validate them. Richer schemas improve interoperability, but they still do not authorize an operation.

TrueFoundry Agent definitions can also carry a response_format, including JSON object or JSON Schema variants, while the Agent Harness runs the model/tool loop. Do not infer from that alone that the Harness automatically performs your preferred business-validation, repair, user-clarification, or postcondition workflow. Those behaviors should be explicit in the agent/application design and tested as such.

Guardrails are another separate layer. TrueFoundry's Gateway documents output guardrails that can reject model responses before returning them, but it also documents an important streaming boundary: LLM output guardrails are skipped for streamed responses because the full response is not available for evaluation. Structured-output conformance, application validation, and guardrails should therefore be reasoned about independently rather than collapsed into one “validation” box.

3. Design the Failure Path First

The happy path is simple. The production question is what happens when the contract is not satisfied—or when it is satisfied structurally but rejected by business rules.

{
  "status": "invalid",
  "schema_version": "refund-intent-v3",
  "errors": [
    {"path": "$.currency", "code": "unsupported_enum"},
    {"path": "$.amount", "code": "exceeds_policy_limit"}
  ]
}

That defect object is more useful than telling a model “your answer was wrong.” It enables targeted repair, metrics by failure class, deterministic fallback, and escalation when repeated attempts hit the same invariant. Keep provider failures distinct from application failures: refusal, incomplete generation, schema incompatibility, structural validation, business validation, authorization, and downstream execution are different failure classes and should not collapse into one retry bucket.

4. Boundaries, Stated Plainly

Structured output reduces interface ambiguity; it does not make a model correct. Schema-conforming nonsense is still nonsense. A portable response_format does not make provider implementations identical. Pydantic validation does not prove facts. A valid tool argument object does not grant authority. And an output guardrail is not a substitute for a schema or business validator.

Not every response should be forced into JSON either. Human-facing explanations are often better as prose with typed state or metadata beside them. The strongest pattern is hybrid: structured state for machines, natural language for people.

References

• TrueFoundry: Chat Completions — Structured Outputs.
• TrueFoundry: Provider support for response schemas.
• TrueFoundry: Instructor integration.
• TrueFoundry: Guardrails and output-validation behavior.
• TrueFoundry SDK types: Agent response formats.
• Model Context Protocol 2026-07-28: Tools, input/output schemas, and structured content.
• OpenAI: Structured Outputs and JSON-mode distinctions.

Schemas establish representation constraints, not truth or authorization. Provider behavior and supported schema subsets vary; business validation, policy enforcement, and failure handling remain separate layers and should be tested independently.

‍