Why tool calling breaks when you switch AI providers

The OpenAI request format ports cleanly between providers. Tool calling does not, and the reason is specific: the format standardizes the shape of a tool call but not the state behind it.

Some providers attach opaque data to each tool call and require it back, unchanged, on the next turn. The OpenAI schema has no field for that data. So it gets smuggled somewhere, and anything in your stack that rewrites the field it was smuggled in silently destroys it. The request still validates. The first turn still works. The second turn returns a 400 that names a field you have never heard of.

This article is the failure modes we hit building a gateway across many providers, and what each one looks like from the outside.

Failure one: the model does not support tools, and says nothing

Start with the simplest, because it wastes the most time relative to its difficulty.

A model without tool support does not reject a request containing tools. It answers the prompt. You get a 200, a well-formed response, and a message.content explaining what the model would do if it could call the function, instead of a tool_calls array doing it.

Nothing in your error handling fires, because nothing errored. Your monitoring is green. The symptom reaches you as "the agent ignores its instructions", which sends you to your prompt, which is the wrong place.

The check is a capability lookup, not a test call. On our catalog:

curl -s https://api.apexapi.dev/v1/models \
  -H "Authorization: Bearer $APEXAPI_KEY" \
  | jq -r '.data[] | select(.type == "chat" and .capabilities.tools == false) | .id'

Four chat models come back today. Two of them are search-preview variants and one is an image model, so the list is not arbitrary, but neither is it guessable from the model name. Anything driving a tool loop should assert on that field at startup rather than discovering it in production.

Failure two: hidden per-call state that must round-trip

This is the one that costs a day, and Gemini is the clearest example.

Newer Gemini models return a thoughtSignature attached to every function call. It is opaque, it belongs to that specific call, and the model requires it back when the conversation returns. If it is missing on the next turn, the request fails with:

Function call is missing a thought_signature in functionCall parts

Now consider what that means for anything speaking OpenAI's format. An OpenAI tool call has an id, a type and a function object. There is no field for provider-specific state. The signature has nowhere to live.

The only field that survives the round trip is the ID, because the ID is the one value clients are expected to echo back verbatim so that tool results can be matched to tool calls. So that is where the signature has to travel. In our Google adapter, the tool call ID is built as a normal identifier with the signature appended in encoded form, and unpacked again when the conversation comes back:

call_a1b2c3d4_0_sig_<base64url-encoded signature>

To a client this looks like an unusually long opaque ID, which is exactly what a tool call ID is supposed to be. It round-trips through any correct client without special handling.

The failure mode is a client that invents its own IDs. Some frameworks regenerate tool call IDs when they rebuild message history, either to normalize them or because a developer assumed the ID was cosmetic. The moment that happens the signature is gone, and the next turn 400s. From inside that framework the error is inexplicable: your tools are valid, your schema is right, and the provider is complaining about a field you never set.

If you take one operational rule from this article: never regenerate a tool call ID. Store it, echo it, and treat it as opaque even when it looks like something you could tidy up. The ID is not always just an ID.

Failure three: dropped tool context, which is worse than an error

When a provider adapter does not implement tool translation, it has two options. Reject the request, or strip the tools and send the rest.

Stripping looks friendlier. It is much worse.

The request succeeds. The model, which was never told a tool exists, answers from the conversation alone. It produces something plausible, because that is what these models do with insufficient context. Your code parses it, finds no tool call, and takes whatever branch it takes when the model chooses not to call a tool. Meanwhile the conversation history now contains an assistant turn that was generated without the tool context every later turn assumes. And you were billed for it.

We hit this decision directly. Our Vertex adapter does not yet translate tool calling into Gemini's function declaration shapes, so it explicitly refuses tool-bearing payloads with a 400 rather than dropping them. The reasoning, written in the code where the decision lives, is that silently dropping tool context corrupts the conversation and still bills the caller for a mangled request. An error is recoverable, because your code can catch it and route elsewhere. A corrupted conversation is not, because nothing tells you it happened.

When you evaluate any provider or gateway, this is worth testing deliberately: send a tool-bearing request to something that does not support tools and see which of the two behaviors you get. It tells you a lot about how the rest of the system will treat you.

Failure four: the same concept, three different shapes

Beyond hidden state, the formats themselves genuinely differ in how tools are declared and how results come back.

In OpenAI's format, a tool is a function object with a parameters schema. The assistant emits tool_calls, and results return as messages with role: "tool" carrying a tool_call_id.

In Anthropic's format, a tool declares an input_schema rather than parameters. Calls and results are content blocks inside messages, tool_use and tool_result, rather than separate top-level fields and roles. A single user message can carry several tool results at once.

Those are not cosmetic differences. Converting between them means splitting one Anthropic user message containing multiple tool_result blocks into several separate tool role messages, in the right order, before any remaining user text. Our compatibility layer does that conversion, which is why an Anthropic-shaped body works on the chat completions path, but the reason it needs doing at all is that the two formats model the same idea with different structure.

The practical consequence: when you switch providers, the conversation history you have already stored may not be replayable in the new format. New conversations work. Resuming an old one hits the mismatch. That is a migration question worth answering before the cutover rather than during it.

Why a gateway should not validate your tool schemas

There is a tempting design where the gateway validates tool definitions before forwarding them. It catches errors early and returns cleaner messages.

We deliberately do not do it, and the reason is compatibility. Different clients send slightly different tool shapes. Editors, agent frameworks and newer SDK versions all have their own dialects, some carrying extra fields, some ordering things differently. A gateway that validates strictly rejects requests the upstream provider would have accepted, and every new client version becomes a support ticket.

So tools and tool_choice pass through untouched. The provider is the schema authority. The trade is that a genuinely malformed schema produces the provider's error rather than a gateway error, which is slightly less friendly and considerably more accurate. You are debugging against the system that actually decides.

A checklist before you switch a tool-using workload

Six things, and they take under an hour on your own traffic.

Confirm the target model supports tools. Read the capability rather than inferring it from the model's reputation.

Run two turns, not one. A single-turn test passes on almost everything. Round-trip state failures only appear when the assistant's tool call and your tool result go back together. Most broken migrations pass the test that was run and fail the one that was not.

Grep your stack for anything that generates tool call IDs. If a framework, a serializer or a piece of your own code constructs IDs when rebuilding history, that is the bug waiting to happen.

Test the unsupported case on purpose. Send tools to something that cannot handle them and see whether you get a 400 or a silent strip.

Test a parallel tool call. Several models can emit multiple tool calls in one turn, and handling one correctly does not prove you handle three.

Check whether stored conversations replay. If you persist history and let users resume, format differences in how tool results are structured will surface there first.

Where this leaves you

Tool calling is the part of the OpenAI-compatible convention that is least standardized, because it is the only part that carries state across turns. Everything else in a chat completion is stateless: you send the whole conversation each time and the shape is the shape. Tool calls break that, because some providers need to recognize their own work coming back.

That is not a flaw anyone is going to fix, since the state is genuinely provider-specific and there is no field in the shared format to carry it. It is a property of the arrangement, and it is worth knowing about before your second turn tells you.

If you are moving a tool-using workload, how to use any AI model through an OpenAI-compatible API covers what else does and does not port, and the errors reference maps the error codes you will see. The model catalog lists tool support per model.