Structured output across providers, and where it fails

Structured output is three different guarantees wearing one name, and most of the confusion when you switch providers comes from not knowing which one you had.

The weakest is asking for JSON in the prompt. No guarantee at all. The middle is JSON mode, where generation is constrained so the result parses, but nothing promises the fields you wanted. The strongest is strict schema enforcement, where the provider validates the output against a schema you supply.

Teams routinely build against the third, migrate to a provider offering the first, and get a 200 response containing a friendly sentence where an object should be. This article is what actually ports, what does not, and the one habit that makes the difference survivable.

The three levels, and why the names do not help

Level one: ask in the prompt. "Respond with JSON matching this shape." The model usually complies. Usually is doing a lot of work in that sentence. You will get markdown fences around the JSON some of the time, a preamble sentence some of the time, and a trailing explanation some of the time. Every production system built this way accumulates a pile of string cleanup before the parse.

Level two: JSON mode. The provider constrains decoding so the output is syntactically valid JSON. That eliminates the fences and the preamble, which is a genuine improvement. It says nothing about which JSON. You asked for {name, email, score} and a valid response containing {result: "unknown"} satisfies JSON mode completely.

Level three: strict schema. You supply a JSON Schema and the provider enforces it during generation. Field names, types and required fields all hold. This is the level people mean when they say structured output, and it is the least uniformly available.

The problem is that these three are marketed with overlapping vocabulary. "Structured output", "JSON mode" and "JSON schema" appear across providers attached to different levels. So the only reliable way to know which one you have is to test it, not to read the feature list.

What happens to the parameter when you switch

There are exactly three behaviors a provider can have for a parameter it does not support, and the difference between them is the difference between a ten-minute fix and a day.

Reject with a 400. Annoying and honest. You find out on the first call. Groq documents this behavior for several unsupported fields, returning a 400 rather than proceeding.

Silently ignore it. The request succeeds, the field is dropped, and generation proceeds unconstrained. Google's Gemini compatibility layer states plainly that any parameter not on its supported list is silently ignored. Your output changes shape and nothing in your monitoring fires.

Forward it and let the model decide. The field reaches the model. What happens next depends on the model rather than on the API layer.

Silent dropping is the expensive one, and it is the reason a migration can look clean in staging and produce malformed records in production a week later. The output is still plausible. It just is not the shape you built the parser around, and if your parser is forgiving, bad records reach the database before anyone notices.

What we do with it, precisely

Worth being exact, since we sit in the middle of this.

For chat completions, we validate the fields we need to reason about for billing and limits, and pass everything else through to the upstream provider untouched. response_format is in the pass-through set. It reaches the provider exactly as you sent it and the provider decides what to do with it.

That is a deliberate choice, and the reasoning is the same as for tool schemas. Clients send slightly different shapes, providers evolve their supported fields, and a gateway that validates strictly starts rejecting requests the provider would have accepted. The provider is the schema authority.

The trade is worth stating rather than hiding: because we forward rather than validate, an unsupported response_format produces whatever that provider does, including silently ignoring it. We do not normalize that away and we cannot, since normalizing would mean either rejecting fields some provider supports or emulating enforcement we do not control.

The second thing to be exact about: our model catalog does not currently advertise a structured-output capability flag. GET /v1/models exposes streaming, vision, tools and image capabilities per model, so you can filter on tool support programmatically. You cannot filter on strict schema support the same way. Until that exists, structured output support is something you establish by testing a model, not by querying it.

Tool calling is the more portable path

This is the practical recommendation, and it comes from the shape of the ecosystem rather than from preference.

A tool definition already contains a JSON Schema. That is what the parameters object is. And tool support is far more consistently implemented across providers than strict response formats are, because tool calling is what agent frameworks depend on and it got attention earlier.

So define one tool whose parameters are the shape you want, force the model to call it, and read the arguments:

extract_person = {
    "type": "function",
    "function": {
        "name": "record_person",
        "description": "Record the extracted person details.",
        "parameters": {
            "type": "object",
            "properties": {
                "name":  {"type": "string"},
                "email": {"type": "string"},
                "score": {"type": "integer", "minimum": 0, "maximum": 100},
            },
            "required": ["name", "email", "score"],
        },
    },
}

response = client.chat.completions.create(
    model="anthropic/claude-opus-5",
    messages=[{"role": "user", "content": bio_text}],
    tools=[extract_person],
    tool_choice={"type": "function", "function": {"name": "record_person"}},
)

call = response.choices[0].message.tool_calls[0]
data = json.loads(call.function.arguments)

You get schema-shaped output on models that offer no strict response format at all, using a mechanism those models already implement well. On our catalog, seventy-six of the eighty chat models advertise tool support, which is a much wider net than any structured-output feature currently spans.

Two caveats so this does not become its own surprise. tool_choice support varies, so check that forcing a specific tool actually forces it rather than merely suggesting it. And the arguments come back as a JSON string that you parse yourself, which means you are validating anyway, which brings us to the part that is not optional.

Validate on your side, always

Whatever level of enforcement you believe you have, parse and validate the result in your own code before it reaches anything that matters. Three reasons, and each one is sufficient on its own.

Enforcement is not uniform. It varies by provider and by model within a provider. The guarantee you tested on one model is not the guarantee you get on another.

Failover moves you. The entire point of a gateway is that a request finds a healthy rail when its first choice is down. If the fallback model has weaker structured output support, your enforcement quietly degrades at exactly the moment everything else is also going wrong. This is the strongest argument for client-side validation and it is the one people forget, because they test the happy path.

Schema-conformant is not correct. A response can satisfy every type and required field and still be wrong. {"email": "unknown@example.com", "score": 0} passes validation and fails your business logic. Schema enforcement buys you shape, never meaning.

The implementation is boring and that is the point. Parse into a typed model with whatever validator you already use, treat a validation failure as a retryable error with a bounded retry count, and log the raw output when validation fails so you can see whether the model is drifting or your schema is wrong. That loop makes structured output reliable across every provider regardless of what each one enforces.

A short checklist before you switch a structured workload

Establish which level you actually have today. Send a request that would satisfy JSON mode but violate your schema, and see whether the provider stops it. If it does not, you are on level two and have been treating it as level three.

Test the unsupported case deliberately. Send response_format to the target and check whether you get a 400, a silent ignore, or honest support. Ten minutes now, one day later.

Try the tool-calling route in parallel. If it works on both your current and target provider, it removes the whole question from your migration.

Confirm your validator runs on the fallback path. Not just the happy path. Force a failover in staging and watch what your parser does.

Log raw output on validation failure. Without it, a drift in model behavior looks identical to a bug in your schema.

If you are working through a provider switch more broadly, how to use any AI model through an OpenAI-compatible API covers what else does and does not port, and why tool calling breaks when you switch AI providers covers the failure modes on the tool path specifically. Tool support per model is in the model catalog, and the errors reference maps the codes you will see.