How AI API failover works, and when it doesn't

On 4 December 2024, a load balancer misconfiguration made OpenAI's API return HTTP 530 to 100% of requests. The total outage lasted about four minutes, from 15:48 to 15:52 PT. A second issue followed, and for the next ninety minutes roughly 45% of API requests still failed. All of it is in OpenAI's own postmortem.

Four minutes is short. It is also long enough that every application with one hardcoded provider and no fallback returned errors to real users for the duration, while applications with a fallback path did not.

That fallback path is what this article is about: how failover detects a provider failure and moves the request somewhere healthy. The idea is simple. The engineering is in the details, and in the part most write-ups skip, which is what failover costs you when it actually fires.

What breaks when a provider fails

Four failure categories, four signatures

Timeouts appear as client-side hangs or HTTP 504 when inference takes too long.

Rate limits arrive as HTTP 429, almost always with a Retry-After header telling you exactly how long to wait.

Provider outages appear as 500, 502 or 503. The diagnostic that matters is not the code but the pattern: they arrive in bursts across many concurrent requests rather than in isolation.

Model deprecation or removal produces model-specific 4xx errors or "model unavailable" messages, which is a different problem from a server failure and needs a different response.

The burst pattern is the signal that separates a provider outage from a bug in your own request. One 500 on one request is noise. A hundred 503s across unrelated requests in the same ten seconds is systemic. That distinction decides whether you retry the same path or escalate immediately, and the threshold depends on your traffic volume rather than any universal number.

Streaming failures are the ones you miss

Server-sent event streaming is the sneakiest failure mode in the list. When a stream breaks mid-generation, the HTTP layer often produces no error at all. The connection closes, tokens stop arriving, and your status-code monitoring sees a clean 200.

The only reliable detection is at the application layer. Watch for a missing terminal marker, an absent finish_reason, or a stream that ends while your JSON accumulator is still mid-object. If a stream ends without a terminal signal, treat it as a failure whatever the status code says. Error handling that only inspects status codes misses streaming outages entirely, and the user sees a truncated answer that looks like a bug in your product.

Detecting failure before it is total

Hard signals and soft signals

Hard signals are unambiguous: error codes, stream aborts, auth failures, schema validation failures. Something failed and you know it.

Soft signals are the useful ones. Sustained p95 and p99 latency spikes, rising retry rates, climbing timeout percentages. These say a provider is degrading before it fails outright.

Hard signals should trigger routing changes immediately. Soft signals should trigger circuit breaker attention. If you watch only error codes, you discover degradation at the same moment your users do.

Circuit breakers

A circuit breaker aggregates repeated failures into an explicit "stop routing here" decision over a rolling window. Three states:

• Closed is normal operation.
• Open means failures crossed the threshold, and requests skip this provider without trying.
• Half-open lets exactly one probe through to check whether it recovered.

This is the mechanism that stops a retry storm from hammering a provider that is already struggling. The threshold is usually either an absolute count or a failure rate over a rolling window, and the rolling part matters: a brief burst of bad requests should not mark a provider unhealthy forever. Only failures inside the current window count.

One detail that bites in practice: the half-open probe needs its own timeout. If the single request that flipped the breaker to half-open never reports back, because the client aborted it, the breaker can sit half-open indefinitely and admit nothing. Ours treats a stale probe as expired for exactly this reason.

Routing the request

Retry first, with backoff

The first tier for any failure is a retry. For a brief timeout, a momentary 429, or an isolated 503, retry the same model with exponential backoff. Flat retries are worse than none, because they add load to a provider that is already under pressure.

Honor Retry-After when a provider sends it. It is a signal rather than a suggestion, and overriding it tends to earn a longer ban.

One prerequisite deserves more attention than it gets: retrying a non-idempotent request produces duplicate side effects. If the request creates a resource or triggers a payment, retry without an idempotency key is its own outage.

Then escalate outward

Once retries are exhausted or the breaker is open, the request escalates through tiers, each one widening the blast radius:

1. Same model, retry. Cheapest, least behavioral change.
2. Same provider, sibling model. Shared tokenizer conventions and tool schemas mean minimal drift.
3. Different provider. Buys independence from a provider-wide outage, introduces the most variance.

Exhaust the cheap tiers before the expensive one.

Weighted routing is the softer alternative to a hard cutover. Instead of moving all traffic off a struggling provider, shift a slice of it. This also keeps the backup path warm, which matters more than it sounds: a fallback that has not seen real traffic in weeks has unknown latency and untested behavior on the day you finally need it.

What failover actually costs you

This is the section most articles skip, and it is where the real decisions live. Failover does not make an outage disappear. It converts a hard failure into a soft one, and the soft one has a price.

Behavioral drift

Different models answer the same prompt differently. Tool schemas diverge, output formatting varies, context-window handling differs. A fallback that technically returns 200 can still break the application, because downstream code was written against the primary model's output shape. Validate the response schema at every tier and reject a malformed fallback before it reaches a user.

Test fallback quality offline, on the same task set production handles, before an incident forces the experiment live. A model that scores badly on your workload is not a fallback. It is a second failure mode with a 200 status code.

A cold prompt cache

This one is almost never mentioned and it is the one that surprises people. Prompt caches are scoped to a provider endpoint. A cache written against one rail does not exist on another.

So the moment failover crosses a rail boundary, the cached prefix you were relying on is gone. Every request pays full input price until a new cache is written on the new rail, and if the incident is brief you may pay the cache write premium twice and read from neither. On a workload with a large stable system prompt, that is a real cost spike arriving at the same moment as the incident.

Spend, and the sleeping engineer

If the fallback tier is pricier than the primary, an outage silently becomes a billing event. Per-tier cost caps exist precisely so a provider incident at 3am does not also become a budget incident. This is the same discipline covered in our post on controlling AI API costs across a team.

Hedging is not free either

Hedging, firing a second request to a backup after a short delay and taking whichever returns first, reduces tail latency. It can also roughly double per-request cost during the hedge window, because both requests may complete and both are billed. Sequential fallback is cheaper when things are healthy, but the user absorbs the full timeout before the second attempt even starts. Neither is universally right.

Where the logic should live

Building failover yourself means owning detection logic, circuit breaker state machines, tiered routing tables, retry policy and a provider abstraction layer, inside your application code. That is the initial build. Then every provider SDK update, model deprecation and error-schema change means retesting it.

The failover layer can also become the thing that breaks. A misconfigured breaker window excludes healthy providers. A stale routing table sends traffic to a deprecated model. This is infrastructure with its own on-call burden, and the burden grows with each provider you add.

At the gateway layer, that logic lives outside your code. On ApexApi, each provider has its own circuit breaker with the closed, open and half-open states described above, and models carry a routing priority that defines the failover order. Your application sends one request to one OpenAI-compatible endpoint; when a rail is unhealthy the gateway skips it. There is nothing to configure in your codebase and nothing to retest when a provider changes.

Worth being precise about one thing, because it is the kind of claim that is easy to overstate: our routing order is a configured priority combined with the circuit breaker gate. It is not a closed loop that automatically re-ranks providers from live scores. Separately, we publish measured success rate and latency per model from real traffic through the gateway, so you can see how rails actually behave and choose your priorities from data rather than vendor claims. Those are two different things and we would rather say so than blur them.

If you want to compare approaches, ApexApi versus OpenRouter and ApexApi versus LiteLLM cover the architectural differences, including the self-hosted option.

Putting the chain together

Detect with hard error signals and soft latency signals, aggregated by a circuit breaker over a rolling window. Route through tiers, closest compatible model first, escalating only as needed. Validate output at every tier so drift does not reach users. Then budget for what failover costs when it fires: drift, a cold cache, higher spend, and duplicated side effects on non-idempotent calls.

The December 2024 incident took four minutes to return errors to every single caller. The question is not whether your provider will have one. It is whether the request has somewhere else to go when it does.

Read the quickstart to route your first request through the gateway, or check the measured rankings to see how the rails have actually performed.