Supporting several model providers sounds like an adapter problem: normalize the request, call an API, and normalize the response. That is necessary, but it leaves the difficult product decisions unanswered.
Which provider is allowed for this tenant? Does it support the required output contract? Can the request cross the configured data boundary? What should happen if the first response starts streaming and then fails?
Describe the job before selecting the model
I want routing to begin with the task. A retrieval rewrite, a customer-facing answer, and a tool-using planning step have different requirements for quality, latency, context size, and output structure.
The request should carry those requirements explicitly, together with tenant policy and a budget. Model selection can then filter out candidates that are not eligible before comparing the remaining options.
I prefer capability checks over provider-name conditionals scattered across the application. A capability record can describe tool support, structured output, streaming behavior, context limits, and the deployment region verified for that configuration.
Keep the shared contract honest
A common adapter should normalize what the application genuinely needs. It should not pretend that every provider implements every feature in an identical way.
For example, tool calls may arrive incrementally during streaming. Usage accounting may arrive at a different stage from text. A provider can reject a schema that another accepts. The adapter needs defined behavior for those differences and tests that exercise the actual wire format.
When a capability is unavailable, I prefer a clear failure or an explicitly approved alternate path. Silently dropping a required constraint creates a response that looks successful while violating the task contract.
Fallback must preserve eligibility
A fallback chain should contain only providers permitted for the current request. Availability does not override a tenant restriction, a region requirement, or a spending limit.
I separate failure classes. A temporary service error may justify another provider. Invalid credentials require a configuration repair. An invalid request may fail identically everywhere. A refusal also needs to be interpreted according to the application’s rules rather than automatically treated as a transport failure.
Each extra attempt consumes time and can consume money. The router needs one total deadline and one attempt budget, not a fresh allowance every time it changes provider.
Decide what partial output means
If no output has reached the user, retrying another provider can be relatively straightforward. Once part of an answer has been displayed, appending a second model’s fresh answer can produce a confusing or contradictory message.
I would either keep generation buffered until the chosen release point or expose an explicit failed response state and offer a restart. The correct choice depends on the product, but it should be deliberate.
Tool execution adds another boundary. A model retry must not repeat a completed external action simply because the next planning step failed. The operation ledger belongs outside the provider adapter.
Measure routing decisions, including degraded service
My routing trace includes task type, eligible candidates, chosen provider, fallback reason, policy version, latency, and accounted usage. Sensitive prompt content does not need to be copied into every metric.
I also separate dependency health by the scope of the failure. A misconfigured internal assistant should not necessarily open the circuit for an otherwise healthy customer chat path. The right circuit-breaker boundary depends on which credentials, models, and deployments share failure conditions.
A multi-provider design is useful when it preserves the product’s promises under changing conditions. The engineering work is making those promises explicit enough that every route, including the emergency one, can be checked.
Updated 25 September 2026.