> For the complete documentation index, see [llms.txt](https://docs.otari.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.otari.ai/otari/for-operators/routing.md). # Routing policies A routing policy is a caller-facing model name that resolves to one or more real models. Use a policy for failover, conditional selection, traffic splitting, spill-over past a rate limit, learned selection, or guardrails the caller cannot remove. Use an [alias](/otari/for-operators/models.md#model-aliases) when one name always maps to one target. ## Failover ```yaml routing: policies: fast: select: - default: openai:gpt-5-mini on_failure: - anthropic:claude-haiku-4-5 ``` Callers send `model: fast`. If the first provider fails before any response is sent, Otari tries the fallback. The response keeps the policy name; Activity and usage record the models that were attempted. In the dashboard, choose Create policy on Routing, then Fallback: name it, pick the main model, and list at least one backup under "If unavailable, use", each a model other than the main one and the other backups. The backups are `on_failure`, in the order shown. Create policy offers one kind at a time (Fallback, Split traffic, Smart router, Spill over at rate limits, Tier down on budget), each with only its own fields. To combine them, open the policy from the list once it exists: its page has a section each for the main selection, the fallback chain, a budget tier-down and guardrails, and each section saves on its own, writing the rest of the policy back unchanged. A policy from `config.yml`, or one using options the page cannot show, opens read-only with its spec in full. ## Selection and failure are separate `select` chooses where a request starts. `on_failure` lists what to try when a candidate fails before its answer starts, whatever the provider's error: an outage, a rate limit, or a request it refuses as invalid (400) all move on. A candidate whose provider has no credential configured fails that way too, so the policy moves on to the next; a provider nothing declares has none, keyless local backends included, and an entry that names no key has none, whatever the provider's own environment variable holds. A candidate nothing declares that one of the deployment's hosted providers serves is sent on the deployment's own credential, as a single model is ([modes](/otari/for-operators/hosted-providers.md#deployment-paid-credentials)). When no candidate has a credential, the request is refused with `424 provider_not_configured`. A candidate whose provider rejects the configured credential (401, or a 403 that does not reject the request itself) moves on too; when every candidate either lacks a credential or had it rejected, and at least one was rejected, the request is refused with `424 provider_credential_rejected`. A Chat Completions candidate that answers with no choices has failed too, so the policy moves on to the next. ```yaml routing: policies: thrifty: select: - when: budget_used_pct: {gte: 80} target: openai:gpt-5-nano - default: openai:gpt-5-mini on_failure: - anthropic:claude-haiku-4-5 ``` Selection entries are evaluated in order. The default must be last. Supported conditions are: * `budget_used_pct` * `budget_remaining_usd` * `user_id` * `key_id` All conditions in one `when` block must match. Numeric comparisons use exactly one of `gt`, `gte`, `lt`, or `lte`. A budget condition does not match a caller with no finite budget. In a policy that can fall over, each candidate gets one upstream call unless a retry count is configured, so a throttled provider hands over to the next candidate at once rather than after its SDK's own retries. See [Provider retries](/otari/for-operators/configuration.md#provider-retries). ## Load balance across providers (weighted routing) The weighted router chooses independently for each request and normalizes the configured weights: ```yaml routing: policies: balanced: select: - router: weighted candidates: - openai:gpt-5 - anthropic:claude-sonnet-4-6 weights: openai:gpt-5: 7 anthropic:claude-sonnet-4-6: 3 - default: openai:gpt-5 on_failure: - gemini:gemini-2.5-flash ``` `7:3` and `70:30` describe the same split. A candidate omitted from `weights` receives no initial traffic but remains available in the failure ordering. At least one weight must be positive. Each request is a fresh draw. Weighted routing has no conversation stickiness or health-adjusted weights. If the selected provider fails before responding, Otari continues through the remaining weighted pool, then `on_failure`. Caller allow-lists filter candidates before weights are normalized. Use `otari routing explain` to see the effective split for a restricted caller. ## Spill over when a model is full (priority routing) The priority router keeps its candidates in the order written. Each request goes to the first one that has room under its [`per: model` rate limits](/otari/for-operators/configuration.md#rate-limit-rules): ```yaml rate_limits: - name: flash-cap per: model models: ["vertex:gemini-2.5-flash"] rpm: 100 routing: policies: summarize: select: - router: priority candidates: - vertex:gemini-2.5-flash # takes every request up to 100 a minute - together:llama-3.3-70b # takes the rest - default: together:llama-3.3-70b on_failure: - mistral:mistral-small ``` A full candidate is skipped without being called, and Activity shows it in the request's routing plan as skipped, naming the limit that was full. Prometheus counts each one in `gateway_rate_limit_model_full{rule, model}`. The limit lives on the model, not on the policy, so it is skipped the same way in `on_failure`, in a weighted pool, and in any other policy that names it; with Redis as `rate_limit_store`, the count holds across replicas. A candidate with room that fails before responding falls through to the next one, as in any policy. The per-model limits refuse a request with a 429 only when every candidate is full, naming the limit that frees up soonest (and its `Retry-After`) but not the model, since a policy's targets are not the caller's to see. A per-key, per-user or deployment rule is checked before routing, so it can still refuse a request while every candidate has room. The policy says which is which: `candidates` handles "full", `on_failure` handles "broke". `Otari-Router: off` skips the order and starts from the default. In the dashboard, choose Create policy on Routing, then "Spill over at rate limits", order the models with the arrows, and add the limit under Settings, Rate limit rules, counted for "Each model". On an existing policy, pick "Spill over" in its Main selection section. The `knn` router uses scored examples to rank candidates for each user's traffic: ```yaml routing: policies: smart: select: - router: knn candidates: - openai:gpt-5-nano - openai:gpt-5 - default: openai:gpt-5 on_failure: - anthropic:claude-haiku-4-5 ``` It embeds the request, finds similar scored prompts, and balances predicted quality against configured model cost. Every candidate therefore needs pricing. Until a user's pool is warm, or when the router cannot decide confidently, the default target serves. Teach the router through `POST /api/v1/routing/preferences/rank`. Each example contains a prompt and a score from 0 to 1 for each candidate. Read pool status through `GET /api/v1/routing/status`. `scripts/seed_routing_demo.py` provides a runnable example. Learned memory is scoped by user and workspace. Optional task IDs create separate pools. It is not learned automatically from live traffic. ### Per-request control | Header | Effect | | ----------------------- | ------------------------------------------------------------------------------- | | `Otari-Router: off` | Skip learned, weighted or priority selection and use the policy default. | | `Otari-Conversation-Id` | Reuse a learned decision for a conversation when granularity is `trace_sticky`. | | `Otari-Router-Task` | Use examples from one task partition. | The default learned-routing settings are `k=5`, a 20-example warm-up, and `trace_sticky` granularity. They are deployment settings named `router_k`, `router_seed_count`, `router_alpha`, `router_confidence_floor`, `router_embedding_model`, `router_granularity`, and `router_max_records_per_user`. Trace stickiness is process-local. A restart or request routed to another replica may choose again. The result is still valid, but prompt-cache locality is not guaranteed across replicas. ## Mandatory guardrails A policy can run guardrails even when the caller did not request them: ```yaml routing: policies: safe: select: - default: openai:gpt-5-mini guardrails: - profile: prompt-injection mode: block on_unavailable: block ``` `mode` is required. `on_unavailable: block` fails closed when the guardrail service cannot run; `monitor` lets the request continue and records the skipped check. Only input checks can be mandated by a policy. Organization-mandated and request-provided guardrails compose with policy guardrails. See [Guardrails](/otari/for-integrators/guardrails.md). ## Explain a policy Inspect a policy without calling a provider: ```bash otari routing explain fast otari routing explain thrifty --budget-used-pct 85 otari routing explain balanced --allowed-model "anthropic:*" ``` The command shows ordered candidates, filtered candidates and their reasons, effective weighted shares, and mandatory guardrails. The API equivalent is `POST /api/v1/routing/policies/explain`; it can also validate an unsaved draft. ## Agent model recommendations A policy decides the model of a request that reaches Otari. A coding agent that talks to its provider directly can still ask Otari which model a new subagent should run on, through `POST /api/v1/routing/recommend`. A self-run deployment puts one choice question to the decision model it configured and recommends the candidate it picks; on otari.ai a managed recommender answers instead. The harness that asks picks the candidates: Claude Code chooses among its own `haiku`, `sonnet` and `opus` aliases, and pi, through [pi-otari](https://github.com/mozilla-ai/pi-otari), among the Otari model ids it sends with the request. Either way the recommendation is billed to the caller, and nothing else is dispatched. See [Use with Claude Code](/otari/for-integrators/use-with-claude-code.md#let-otari-choose-a-subagents-model). ## Managing policies at runtime Config-file policies apply to every workspace. Operators can also manage stored policies through the Routing page or `/api/v1/routing/policies`. Stored policies belong to one workspace and can optionally be scoped to one user. The operator's list, like `/api/v1/aliases`, shows the stored entries of the workspaces in the operator's active organization, and the master key acts in the default organization. An organization's owners and admins manage their own workspaces' policies and aliases through `/api/v1/organizations/me/routing-policies` and `/api/v1/organizations/me/aliases`, which the Routing page uses for a caller who does not operate the deployment. Those routes require the workspace named, must name a workspace of the caller's own organization, accept no user scope, and refuse a target the organization holds no provider access for. Both reads take a server-capped `limit`, which bounds the response rather than paginating it. A member of the organization reads the same two lists and writes neither. Those routes are workspace-wide entries only, on the reads as much as the writes. A stored policy or alias can also be scoped to one user, and `user_id` names an API-key user rather than a dashboard identity, so it is neither a tenant's to set nor a tenant's to interpret. Managing a user-scoped entry stays on the deployment-wide routes. The authenticating API key determines which workspace resolves a policy. User-scoped entries take precedence over wider entries. A config-file policy cannot be changed through the API. Changes apply immediately on the worker that accepts the write. Other workers refresh stored routing configuration within 30 seconds. Use the generated OpenAPI document for create, rename, delete, list, and explain request schemas. ## Rules and limits * A plan may contain at most five candidates, including router pools and `on_failure`. * Targets must be concrete provider or instance selectors. Policies and aliases cannot chain. * Policy names cannot contain `:` or `/` or collide with an alias or provider instance. * `routing.enabled: false` disables policy resolution but still validates the configured policies. * Caller model allow-lists apply to every candidate. * Dynamic policies apply to Chat Completions, Messages, and Responses. A static one-target policy can resolve on other model-taking endpoints. ## What is billed and what the caller sees Pricing, budgets, and usage use the resolved model. Completion responses use the policy name. Each failed attempt before a successful fallback gets an `absorbed` usage row. All attempts share a `request_group_id`, which is the request's `Otari-Request-ID`. Absorbed rows have no settled model cost and do not increase request or error totals; the final row represents the caller-visible request. Every successful Chat Completions, Messages, and Responses response, routed or not, also says how it was served, so a caller's own metrics can see a fallback without reading the usage log: | Header | Value | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `Otari-Provider` | The provider instance that served the request, by its configured name (`openai`, `azure-eu`). Never its base URL or credentials. | | `Otari-Model` | The model the serving provider was sent, as that provider names it (`gpt-5-mini`, `claude-haiku-4-5`), without the instance prefix. A routing policy, an alias, or a catalog ID keeps its own name in the body's `model`, so this is where a caller sees which model answered. A request naming a model reports that model. Left out when the name is not printable ASCII. | | `Otari-Attempt-Count` | How many candidates the request was sent to, including the one that served. A candidate skipped without being called, such as a model a rate limit had no room on, is not counted. | | `Otari-Fallback` | `true` when a candidate other than the plan's first served the request, whether the first failed or was skipped; otherwise `false`. | | `Otari-Response-Duration-Ms` | Whole milliseconds from the start of the handler preamble to the provider answering the request, failed candidates included. For a streamed response, to the stream opening or its first chunk, since the headers go out then. | | `Server-Timing` | The same duration split into Otari's time and the provider's, in [W3C Server Timing](https://www.w3.org/TR/server-timing/) form: `otari;dur=4.2;desc="Otari before the provider", upstream;dur=812.5;desc="Provider until its answer or first chunk", total;dur=816.7;desc="Otari and the provider"`. `otari` runs until the first candidate is sent: the budget hold, routing, rate limits and preparing the call. `upstream` runs from there to the answer, failed candidates included, or for a stream to its first chunk, or to the end of the first-chunk hold when the provider is slower. Gateway tools that run between provider calls, such as web search or code execution, count toward `upstream`. A stream that uses gateway tools opens before its first provider call answers, so for one of those `upstream` and `total` end when the stream opens, as `Otari-Response-Duration-Ms` does. Durations are milliseconds to a tenth. | A request naming a model or an alias reports `1` and `false`. Streaming responses carry the same values: failover happens before the stream opens, so the serving candidate and the attempt count are already known when the headers are sent. A failure later in the stream is reported in the stream and changes neither. The request's cost is `usage.cost_usd` in the body, as described in the [API reference](/otari/for-integrators/api-reference.md#request-id-and-inline-cost). Built-in tool charges settle on the final row. Candidate price and remaining budget, including a budget narrowed to that candidate's provider, are checked before each attempt, so a fallback cannot silently bypass pricing or spend limits. A fallback is skipped when a budget narrowed to its provider has no room, and the next candidate is tried. A budget on every provider that cannot cover a pricier fallback stops the chain instead. What a fallback spends counts against its own provider's budgets. ## Failure behavior Failover occurs only before response bytes reach the client. A streamed candidate that fails before its first chunk falls over like one that fails to open, as long as the failure comes within `streaming_first_chunk_hold_ms` (15 seconds by default; `0` limits failover to candidates that fail to open). A streaming failure after the stream begins is returned to the client because switching models mid-answer would corrupt the response. A tool loop that has already produced assistant state cannot be replayed on another provider. On Chat Completions, a stream that has reported one of Otari's web searches or fetches in `otari_tool_calls` has sent response bytes, so it is committed from that report on. Shared service failures, such as an unavailable mandatory guardrail or sandbox, also do not improve by changing candidates. If every candidate of a plan with more than one candidate fails, Otari returns a gateway error in its own words, never a provider's message. A plan with one candidate answers as naming that model directly would. When every candidate rejected the request as invalid (400), Otari returns 400 with the code `all_candidates_rejected`, because the request is at fault. When every candidate found the prompt too long, the code is `context_length_exceeded`. When no candidate knew the model (404), Otari returns 502, because Otari chose the candidates. If caller restrictions remove every candidate, Otari returns 403 without revealing the hidden targets. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.otari.ai/otari/for-operators/routing.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.