> For the complete documentation index, see [llms.txt](https://docs.otari.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.otari.ai/otari/for-operators/configuration.md). # Configuration Otari reads a YAML file, environment variables, and selected settings stored by the management API. Start with [`config.example.yml`](https://github.com/mozilla-ai/otari/blob/v0.33.0/config.example.yml); the running dashboard's Settings page shows the effective non-secret scalar configuration and which values can be changed without a restart. ## Config file Pass a file explicitly: ```bash otari serve --config config.yml ``` A small configuration looks like this: ```yaml database_url: "postgresql://otari:otari@postgres:5432/otari" master_key: ${OTARI_MASTER_KEY} default_pricing: true providers: openai: api_key: ${OPENAI_API_KEY} ``` String values support `${ENV_VAR}` interpolation. Keep credentials in the environment or a secret store rather than committing them to YAML. A reference to a variable that is not set fails the load. A variable that is set but empty leaves a `providers` entry with no credential, and Otari ignores such an entry with a warning that names the variable, so a credential stored for the same provider through the dashboard serves its requests instead of being shadowed by the empty one. ## Environment variables Every scalar `GatewayConfig` field can be overridden as `OTARI_`, for example: ```bash export OTARI_DATABASE_URL="postgresql://otari:otari@localhost:5432/otari" export OTARI_MASTER_KEY="..." export OTARI_DEFAULT_PRICING=true ``` Provider credentials come from the environment through `${VAR}` references in the `providers` map (see [Provider configuration](#provider-configuration)). A provider's native variable, such as `OPENAI_API_KEY`, is never read on its own: name it in the entry (`api_key: ${OPENAI_API_KEY}`); see [Models](/otari/for-operators/models.md#configuring-a-provider). Booleans accept `true`, `false`, `1`, `0`, `yes`, `no`, `on`, and `off`, without regard to case. ### Full config via environment Container platforms can supply the whole YAML document through `OTARI_CONFIG_YAML`, or its base64-encoded form through `OTARI_CONFIG_B64`. Use only one; raw YAML wins when both are present. ```bash export OTARI_CONFIG_YAML=' default_pricing: true providers: openai: api_key: ${OPENAI_API_KEY} ' ``` Precedence from lowest to highest is the config file, structured environment config, then `OTARI_` values: the scalar fields, and `OTARI_HOSTED_PROVIDERS` and `OTARI_PROVIDER_SUPPORTED_COUNTRIES` as JSON. A top-level key of the structured config, and those two variables, replace the lower layer's value whole rather than merging into it. Stored runtime settings override the corresponding startup value after the database is available. ## Common settings | Setting | Purpose | | -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `database_url` | SQLite or PostgreSQL connection. PostgreSQL is recommended for production. | | `master_key` | Deployment-wide management credential. | | `host`, `port` | Server bind address. | | `auto_migrate` | Apply Alembic migrations at startup. | | `require_pricing` | Reject unpriced, budgeted traffic. Defaults to `true`. | | `default_pricing` | Use the models.dev price snapshot when no stored price exists. | | `pricing_refresh` | Whether the catalog updates itself on a schedule, and whether its price changes wait for review: `manual`, `review`, or `auto`. | | `feedback_enabled` | Allow deliberate feedback submissions to the Otari team. Defaults to `true`; startup setting. See [Product feedback](#product-feedback). | | `health_checks_enabled` | Offer the Health page and its scheduled checks of the hosted providers' models. Defaults to `true`; startup setting. See [Health checks](/otari/for-operators/health-checks.md). | | `public_catalog` | Serve the model catalog to visitors without a session. Defaults to `false`. | | `public_catalog_rate_limit_per_minute` | Anonymous catalog reads per client address per minute. Defaults to 60. | | `rate_limit_rpm` | Per-user request limit. Unset disables it. | | `rate_limit_store` | Where `rate_limit_rpm` and the `rate_limits` rules are counted: `memory` (the default) or `redis`. See [Rate limits across replicas](#rate-limits-across-replicas). | | `rate_limit_redis_url` | The Redis that the `redis` store counts in. | | `rate_limits` | Requests per minute, tokens per minute and requests in flight, per deployment, API key, user or model. Also managed from the dashboard. See [Rate limit rules](#rate-limit-rules). | | `budgets` | Budgets declared by id and written to the database at every start. See [Budgets and API keys in config](#budgets-and-api-keys-in-config). | | `api_keys` | API keys, service keys among them, declared with a secret you choose and written to the database at every start. See [Budgets and API keys in config](#budgets-and-api-keys-in-config). | | `federation_issuers` | OpenID Connect issuers whose identity tokens may authenticate as an API key in place of its secret. Empty, the default, turns it off. Setting it requires `OTARI_SECRET_KEY`, or the gateway refuses to start. See [Workload identity](/otari/for-operators/workload-identity.md). | | `federation_member_rules` | Let an organization's owners and admins give their own API keys federation rules naming any public https issuer, matched only in the token exchange that names them. Defaults to `false`. Setting it requires `OTARI_SECRET_KEY`. See [Rules on a member's own key](/otari/for-operators/workload-identity.md#rules-on-a-members-own-key). | | `federation_allow_private_hosts` | Let a federation issuer's discovery document and signing keys be fetched from a private address. Defaults to `false`. | | `federation_access_token_ttl_sec` | How long an access token from the workload identity token exchange lasts, in seconds. Defaults to 600; between 60 and 3600. See [Claude Code and the Anthropic SDKs](/otari/for-operators/workload-identity.md#claude-code-and-the-anthropic-sdks). | | `provider_max_retries` | How many times a provider SDK retries a failed call before Otari sees the failure. Unset keeps each SDK's default. See [Provider retries](#provider-retries). | | `provider_supported_countries` | The countries each provider type serves, such as `openai: [US, DE]`. A request from another country skips that provider. Empty, the default, turns it off. See [Supported countries](#supported-countries). | | `client_country_header` | A header a trusted proxy sets to the client's country code, such as `CF-IPCountry`. See [Supported countries](#supported-countries). | | `client_country_database` | Path to a MaxMind DB country file the client address is looked up in. When set, a country file stored from the Settings page is not read. Needs the `geoip` extra. See [Supported countries](#supported-countries). | | `streaming_keepalive_interval_ms` | Idle interval after which a streaming response sends a keepalive, so an intermediary with a read timeout does not cut a slow stream. Defaults to 15000; `0` disables keepalives. | | `streaming_first_chunk_hold_ms` | How long the gateway holds a streaming response's status line for the provider's first chunk, so a provider failure before any output gets its own HTTP status, and falls over in a routing policy, rather than ending a `200` in an error event. A slower provider still streams. Defaults to 15000; `0` commits the response as soon as the stream opens. See [Error codes](/otari/for-integrators/api-reference.md#error-codes). | | `idempotency_retention_sec` | How long a completion sent with an `Idempotency-Key` is kept for a retry to replay. Defaults to a day; `0` ignores the header. Needs `OTARI_SECRET_KEY`, which encrypts the stored response. See [Retrying safely](/otari/for-integrators/api-reference.md#retrying-safely). | | `provider_request_timeout_seconds` | How long a non-streaming provider call may take, in seconds. Defaults to 600. Connecting stays capped at 5 seconds for providers whose SDK separates the two (Anthropic, OpenAI and OpenAI-compatible, Groq, Cerebras, Together); Bedrock, Gemini, Vertex AI, Mistral and Cohere apply the value as a whole. Without it, Anthropic's SDK refuses a non-streaming request with a large `max_tokens`. Providers with no per-request timeout, and a provider whose `client_args` sets its own `timeout`, keep their client's timeout. | | `enable_metrics` | Serve Prometheus metrics at `/metrics`. Needs the `metrics` extra (`pip install gateway[metrics]`), which the Docker image installs; setting this without it refuses to start. | | `accept_incoming_trace_context` | Join spans the gateway creates to the caller's trace. Defaults to `false`. See [Trace context propagation](#trace-context-propagation). | | `enable_docs` | Serve OpenAPI, Swagger UI, and ReDoc. | For every field, its current default, validation, and description live on `GatewayConfig` in `src/gateway/core/config.py`, or in the per-domain module under `src/gateway/core/settings/` that it inherits. Operators can read the non-secret effective set through `GET /api/v1/settings`. ### Database connections | Setting | Purpose | | --------------------------------- | ------------------------------------------------------------------------------------------------------- | | `db_pool_size`, `db_max_overflow` | Connections available for serving requests. | | `db_pool_timeout` | Seconds a request waits for a free connection before it is refused. | | `db_pool_recycle` | Retire a connection after this many seconds. `-1` disables. | | `db_connect_timeout` | Seconds to wait for a new connection to be established. | | `db_command_timeout` | Client-side ceiling on one statement. `0` disables. | | `db_statement_timeout_ms` | Server-side ceiling on one statement, the backstop for the setting above. Must exceed it; `0` disables. | | `db_log_pool_size` | Connections reserved for usage logging, separate from the pool above. | The pool sizes bound concurrent *database* work, not concurrent provider calls: a request hands its connection back before the upstream call. Raise them for a deployment whose dashboard and management traffic are heavy, not because inference is. Recycling and the two statement timeouts matter most behind a managed database or a NAT, which drop idle connections without closing them. The pool pings a connection that has sat idle for more than five seconds before handing it out, which would catch a closed one, but the ping is itself a statement and blocks on a socket that went away silently, so leaving these unset turns a dropped connection into a request that hangs for minutes. `db_command_timeout` is enforced client-side and `db_statement_timeout_ms` server-side, so the second one still ends a statement when the client is the stuck half. Configure the server-side value above the client-side one, which the defaults do and startup validation requires: set equal, whichever fires first is a race. ### Rate limits across replicas `rate_limit_rpm` is counted in this process by default, so a deployment running N replicas (or N workers) admits up to N times the limit. To hold the limit for the deployment as a whole, count it in Redis: ```yaml rate_limit_rpm: 600 rate_limit_store: redis rate_limit_redis_url: redis://redis:6379/0 ``` This needs the `redis` extra (`pip install gateway[redis]`), which the Docker image installs, and Redis 5 or later. Startup refuses `redis` without a URL or without the extra, because quietly counting per process would multiply the limit again. Each request is checked and counted in one step, against Redis's own clock, so replicas never both take the last slot and their clock skew does not matter. If Redis cannot be reached, each replica counts on its own instead of refusing traffic, and tries Redis again a few seconds later; the gateway log says when that starts and stops. ### Rate limit rules `rate_limits` sets limits beyond `rate_limit_rpm`. Each rule names what one count is shared by (`per`) and sets any of three limits: ```yaml rate_limits: - name: keys # each API key on its own per: key rpm: 600 tpm: 200000 - name: end-users # each user, including a service key's end users per: user rpm: 30 - name: everyone # one count for the whole deployment per: deployment max_concurrent: 200 - name: flash-cap # each model listed, however a request reaches it per: model models: ["vertex:gemini-2.5-flash"] rpm: 100 - name: feature # every user on one budget together, each budget listed on its own per: budget budgets: ["", ""] rpm: 600 mode: report # served anyway, and named in Otari-Rate-Limit-Exceeded ``` * `rpm`: requests per minute. * `tpm`: tokens per minute, counted on what each request used. A request is admitted while the minute's tokens are under the limit and charged what it used once it completes, as LiteLLM does, so the last request admitted in a minute can take the count past the limit. A request that fails is charged the tokens its provider reported before failing, usually none, and a request refused after admission (by its budget, say) is charged nothing. `tpm_admission: estimate` counts the way providers count their own quotas instead: a request holds an estimate (its prompt plus `max_tokens`, or `budget_estimate_default_output_tokens` when it sets none) while it runs, and is refused when that does not fit. Use it for a limit meant to stay under a provider's quota, such as a `per: model` rule that should move traffic to the next model before the provider refuses it. Under it, a request whose estimate alone exceeds the limit is always refused, with no `Retry-After`, since waiting would not let it in. * `max_concurrent`: requests in flight at once. A slot is given back when the response ends, streamed or not. `lease_sec` (15 minutes by default) bounds how long a slot outlives a process that dies holding it. `per: key` does not limit a request made without an API key (the master key or a dashboard session), and `per: user` does not limit one billed to no user. A request has to fit every rule that applies to it; one that does not is refused with a 429 naming the rule and the limit it hit, counted by none of them, and holds no budget. Rules count in `rate_limit_store`, so with Redis they hold across replicas. They apply to chat completions, messages, responses and search, after `rate_limit_rpm`. `per: model` limits each model in `models`, written as `instance:model` (the provider instance it is called through, then the model). One count is kept per model and shared by every policy, alias and direct call that reaches it, so it can stand for a provider's quota. It is checked when an attempt is about to call the model, after any policy has picked its candidates, rather than at admission: a [routing policy](/otari/for-operators/routing.md#spill-over-when-a-model-is-full-priority-routing) skips a full model and tries its next candidate, and a request is refused with a 429 only when no candidate has room. A direct call to a full model is refused. A candidate that fails before responding gives back its tokens and its slot but keeps its request counted, since the provider was sent it. A search tool is limited the same way, named as `provider:tool` (`exa:exa-search`). `per: budget` keeps one count for every user on the same budget, such as all of a service key's end users on one feature's budget, so it limits a feature as a whole where `per: user` limits each user in it. `budgets` narrows it to the budgets listed by `budget_id` (as `GET /api/v1/budgets` returns it), each still counted on its own; unset, it counts every budget. A user on no budget is not counted by it, and neither is a request from a key with `exclude_from_budget` or any request while `budget_strategy` is `disabled`. Like `per: deployment`, it is counted after the rules that count the caller alone, so a request over its own limit takes no place in a count other users share. `mode: report` counts a rule the same way and refuses nothing: a request over it is served, routing still picks the same candidate, and the response carries `Otari-Rate-Limit-Exceeded` naming every report rule it was over (comma-separated). The gateway also logs it and counts it in `gateway_rate_limit_reported_total` by rule. A request something else refuses is not reported. A reported request is not counted in the window it was over, as a refused one is not, so the count is the one the rule would keep if it were enforced. Use it to watch a limit before enforcing it, or to see when traffic borrows beyond a share without refusing anyone; `mode: enforce` is the default. Rules can also be added, changed and removed from the dashboard (Settings, Rate limit rules) or through `/api/v1/rate-limits`. A change applies at once on the replica that served it and on every other replica within 30 seconds; a changed rule keeps the requests it already counted. Choose "Each model" to add a `per: model` rule and pick its models. `per: budget`, `budgets` and `mode` are set through the API. The rules in config.yml are listed there read-only, and a stored rule cannot take the name of one. A stored rule whose name config.yml later declares is skipped with a warning at startup and left out of the list; `DELETE /api/v1/rate-limits/{name}` still removes it. ### Trace context propagation With `accept_incoming_trace_context: true`, Otari extracts incoming context with OpenTelemetry's configured propagators (`OTEL_PROPAGATORS`; W3C `traceparent`, `tracestate` and `baggage` by default) and makes it current for the request, so the spans the gateway creates join the caller's trace. The context is detached when the response, including a streamed body, finishes. Missing or invalid headers never reject a request; the gateway starts a new trace instead. Otari does not inject propagation headers into provider or platform requests. It is off by default because the headers are unauthenticated: the middleware runs before route auth, so any caller can choose the trace ID and sampling flag your collector ingests, and a malformed `tracestate` makes OpenTelemetry log a warning per offending member (`opentelemetry.trace.span`). Enable it for trusted service-to-service callers, ideally behind a proxy that strips these headers at the edge. Browsers cannot send them cross-origin: they are not in the CORS allow-list. ## Budgets and API keys in config A deployment can declare its budgets and API keys in config.yml instead of creating them through the management API. Each start creates the ones missing from the database and writes back the ones whose entry changed, so a service that calls Otari needs only this file and one secret that both sides read from the environment. ```yaml budgets: end-user-budget-ai: max_budget: 0.1 reset_alignment: calendar_day rpm_limit: 40 tpm_limit: 2000 mlpa-global: max_budget: 100 reset_alignment: calendar_day api_keys: mlpa: secret: ${MLPA_SERVICE_KEY} user_id: mlpa is_service_key: true end_user_budget_ids: [end-user-budget-ai] end_user_budget_id: end-user-budget-ai ceiling: mlpa-global ``` A budget is keyed by its id and takes the fields `PUT /api/v1/budgets/{budget_id}` takes: `name`, `max_budget`, `token_limit`, `request_limit`, `rpm_limit`, `tpm_limit`, and either `budget_duration_sec` or `reset_alignment`, not both. An id is up to 128 letters, digits, `.`, `_` and `-`, and starts with a letter or digit. A key is keyed by a name of the same form, which is how Otari finds it again at the next start. It takes: | Field | Meaning | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `secret` | Required. The secret requests present, interpolated from the environment. | | `key_name` | Display name. Defaults to the key's name in config. | | `user_id` | The user the key bills to, created when missing. Unset uses the shared `default` user. | | `is_service_key` | Lets requests name end users. See [Service keys and end users](/otari/for-operators/access-control.md#service-keys-and-end-users). | | `end_user_budget_ids` | Budgets a request may start a new end user on with `Otari-End-User-Budget`. | | `max_end_users` | Most live end users the key may have on the budgets it assigns, or on any budget when it assigns none. See [Capping a key's end users](/otari/for-operators/access-control.md#capping-a-keys-end-users). | | `end_user_budget_id` | Budget a new end user starts on when the request names none. Must be on `end_user_budget_ids` when both are set. | | `ceiling` | Budget that caps the key as a whole, across all of its end users: a scoped budget with `scope_type: api_token` on the key. | | `exclude_from_budget` | Log the key's cost without enforcing any budget. | | `reject_user_mismatch` | Per-key override of the deployment setting. | Unknown fields are refused, so a misspelled limit stops the start rather than leaving a budget uncapped. A key may name a budget declared here or one that already exists in the database. ### The secret The secret is yours to choose, so the same value works across restarts and replicas, and rotating it is a matter of changing the variable on both sides. It must look like a key Otari mints: `tk-` followed by at least 47 letters, digits, `-` and `_`, random enough not to read as a placeholder. Generate one with: ```bash python -c "import secrets; print('tk-' + secrets.token_urlsafe(48))" ``` Otari refuses to start when a secret is empty, is a placeholder, does not have that form, equals the master key, or is shared by two keys. Only its hash is stored, and no log line, error or API response carries it: the keys API shows its usual prefix and last four characters. ### What config.yml owns Config is the source of truth for what it declares, and a start writes an entry back whenever the entry changed: * A missing budget or key is created. An existing one gets the declared values when its entry in config.yml differs from the one a start last wrote over it, so a change made through the API or the dashboard, such as lowering a budget or deactivating a key in an emergency, lasts across restarts and new replicas until config.yml changes that entry. Changing any field of the entry writes the whole entry back: a key that was deactivated is active again. Fields a key does not declare (expiry, allowed models, metadata) are left as they are. * Changing a key's secret in config, including through the environment variable it reads, changes its secret at the next start, and the old one stops working. * `end_user_budget_ids`, `end_user_budget_id` and `ceiling` are part of the key's entry and are reconciled with it. End users already created keep the budget they started on, as they do when the key is changed through the API. A ceiling moved to another budget restarts its period and keeps the spend it has recorded; a ceiling already on the declared budget keeps its period, so a restart does not reset it. A key with no `ceiling` keeps any ceiling it has, and an unchanged entry leaves a ceiling changed or removed through the API as it is. * Otari keeps a digest of each entry it wrote beside the budget or key, never the secret itself (the secret enters as its hash), to tell whether the entry changed. The first start of a version that keeps it writes every entry once, as every start used to. * Nothing is deleted. A budget or key removed from config stays as it is and stops being marked as declared, so it can then be changed or deleted through the API. Declaring it again writes the entry back over it at the next start, even an entry unchanged from before. A key deleted through the API while config still declares it is created again at the next start. `GET /api/v1/budgets` reports `origin: config` on a declared budget, and `GET /api/v1/keys` reports the key's `config_name`. A key whose secret is already on a key minted through the API takes that key over, keeping its id, its end users and its ceiling, which is how an existing key moves into config. Every replica applies the declarations when it starts. They are written in one transaction, and each insert yields to a row another replica placed first, so replicas starting together create one budget, one key and one ceiling. A declaration that cannot be applied, such as a key naming a budget that does not exist, stops the start and nothing is written. Declarations are applied before the first-run bootstrap key, so a fresh deployment that declares a key does not mint and print one. ## Provider configuration > **Deprecated.** The `providers:` block is being retired in favor of hosted providers, which the Hosted providers page and `/api/v1/hosted-providers` manage. At startup, each entry a hosted provider can hold is copied onto a hosted provider of the same name, once: its `${VAR}` values are read at that start, and the hosted provider serves the name from then on, with a serving switch, a test, its models and removal like any other. The start asks the provider for its models and switches on every one it lists, priced or not, and a later refresh keeps them on; one that lists none is not copied, and keeps serving from `config.yml`. The start then logs that the entry is no longer read, so it can be taken out of `config.yml`; rotate its key on the Hosted providers page rather than in the environment. Removing the copied provider on the page sticks: the entry is not copied again. An entry a hosted provider cannot hold keeps serving from `config.yml`, and the start logs why: one that sets `models`, `max_retries`, `messages_api` or Vertex AI's `credentials`, `project` or `location`, a keyless Bedrock or SageMaker entry served on the server's own cloud role, a name that names another provider or that a hosted provider already holds, or a key with no `OTARI_SECRET_KEY` to encrypt it. Once copied, an organization's own key for the provider is asked first, where the entry was asked before it. Set `import_configured_providers: false` (`OTARI_IMPORT_CONFIGURED_PROVIDERS`) to keep serving every entry from `config.yml` for now. The `providers` map is keyed by provider instance. Only a provider it declares (or a hosted provider, or an organization's key) serves requests. A standard provider needs only its credential, which the entry names: ```yaml providers: anthropic: api_key: ${ANTHROPIC_API_KEY} ``` A keyless local backend or a provider authenticating from ambient cloud credentials is declared with a bare entry: ```yaml providers: ollama: bedrock: ``` A custom or self-hosted endpoint can use a named instance: ```yaml providers: home_lab: provider_type: openai api_base: "https://models.example.com/v1" api_key: ${HOME_LAB_TOKEN} models: [qwen3-32b] ``` Call it as `home_lab:qwen3-32b`. The optional `models` list supplies discovery for backends without a model-listing endpoint. See [Models](/otari/for-operators/models.md). ### Provider retries Most provider SDKs retry a failed call themselves before Otari sees the failure. The OpenAI and Anthropic SDKs, and every OpenAI-compatible provider built on them, retry a 429, a 5xx or a dropped connection twice by default and wait out the provider's `Retry-After`, so one request to a throttled provider can make three upstream calls and take many seconds to fail. Google's SDK (Gemini, Vertex AI) does not retry by default. A client that retries on top of that multiplies the calls again. `provider_max_retries` sets the retry count for every provider instance, and an instance's own `max_retries` overrides it: ```yaml provider_max_retries: 0 # OTARI_PROVIDER_MAX_RETRIES providers: openai: api_key: ${OPENAI_API_KEY} max_retries: 1 ``` When neither is set, each SDK keeps its own default, except in a routing policy that can fall over to another candidate (an `on_failure` chain or a weighted or router pool): there every candidate makes one upstream call, so a failure moves to the next candidate at once instead of after the SDK's retries. Setting either value restores retries inside such a policy too. The setting reaches the OpenAI-compatible, Anthropic (including Azure and Vertex AI), Groq, Cerebras, Together, Gemini and Vertex AI clients. Other providers (Bedrock, Mistral, Cohere, Ollama and the rest) keep their SDK's own retry behavior, which `client_args` can still configure where the SDK allows it. A retry setting in an instance's `client_args` (`max_retries`, or `http_options.retry_options` for Google) wins over both values. Neither value reaches an [owned provider endpoint](/otari/for-operators/provider-endpoints.md), which keeps its SDK's default outside a policy that can fall over. ### Provider-specific request fields A chat completion request field that Otari does not know is dropped before the call, so a parameter only one provider takes never reaches it. An instance can let named fields through with `request_fields`. Exa's chat endpoint, for example, takes `text` to return the full text of its sources: ```yaml providers: exa: provider_type: openai api_base: https://api.exa.ai api_key: ${EXA_API_KEY} request_fields: [text] ``` A request reaching that instance with `"text": true` sends it to Exa in `extra_body`, which the OpenAI SDK, and the OpenAI-compatible providers built on it, merge into the request body. A field the instance does not list is dropped as before, and so is every listed field on a request another instance serves, including a routing policy's other candidates. A name the request already carries (`temperature`, `user`, `safety_identifier`, ...), that shapes it (`model`, `messages`, `stream`) or that Otari sets itself (a credential, `api_base`, `extra_body`) cannot be listed, so a caller can never override the gateway's own values through it. Chat completions only. ### Messages through the Responses API A request to `/v1/messages` for a model with no Anthropic Messages API of its own is served as a chat completion by default. An OpenAI instance (or any instance whose provider has a Responses API, such as Azure OpenAI) can serve it through the Responses API instead: ```yaml providers: openai: api_key: ${OPENAI_API_KEY} messages_api: responses # default: completions ``` This reaches models OpenAI serves only on the Responses API (the codex and pro variants), and it keeps a reasoning model's reasoning across turns: each reasoning item comes back as a `thinking` block whose signature carries the item's encrypted content, and when a client such as Claude Code sends that block back on the next turn, Otari hands the reasoning back to the provider. The reasoning is encrypted for the provider account that produced it, so the signature also names the instance that served it: only that instance gets the reasoning back, and any model it serves can use it. A thinking block signed this way is dropped before a call to any other instance or provider, and another provider's thinking blocks are dropped before a Responses call. When the block it drops opened an assistant turn whose tool calls the request answers, `thinking` is also left off that call, because Anthropic refuses a tool-use turn continued under thinking without one; a route that falls back from such an instance to an Anthropic model mid-loop therefore answers that one turn without extended thinking. Reasoning fields are sent only to a model whose name looks like a reasoning model (`o1`, `o3`, `o4-mini` and the rest of the `o` series, `gpt-5*` except its `-chat` variants, and anything with `codex` in it), whether or not the client asked for thinking, because any other model fails the call over them. An Azure OpenAI deployment is called by its deployment name, so name a reasoning deployment after its model (`gpt-5-mini`, not `prod-reasoner`) or its reasoning is neither requested nor carried across turns. Every call is sent with `store: false`, so nothing is kept upstream. A Messages `thinking` budget becomes a Responses `reasoning.effort`: up to 2,048 tokens is `low`, up to 16,384 is `medium`, and more is `high`. An `output_config.effort` takes precedence (`max` becomes `high`). Reasoning summaries are requested unless thinking is disabled. `stop_sequences` is refused with a 400, because the Responses API has no equivalent; `top_k`, `cache_control` and `metadata` are dropped (OpenAI caches prompts on its own). When the caller sends no `prompt_cache_key`, the request's session keys the provider's prompt cache, so the turns of one conversation land on the same cache: `Otari-Conversation-Id`, or the session id Claude Code sends on every request. The key is scoped to the caller like any prompt cache key, so the session id itself never reaches the provider. Anthropic's server tools are not forwarded: Otari's own web search, web fetch and code execution serve them, as for every non-Anthropic model. The setting covers `/v1/messages` traffic only, and only where the instance's provider has a Responses API; an instance whose provider has none keeps the default and logs a warning once. Any other value fails at startup. It is read from the `providers:` block of the gateway's own config. ### Bedrock read timeout A Bedrock client waits up to 600 seconds for the next part of a response. botocore's own default is 60 seconds, which fails a streamed response whose model goes quiet for a minute, as a model writing a long tool call can. A `config` in the instance's `client_args` takes the botocore `Config` options and wins over the default: ```yaml providers: bedrock: client_args: config: read_timeout: 120 ``` ### Runtime provider management Operators add provider credentials at runtime as hosted providers, on the Hosted providers page or through `/api/v1/hosted-providers` (see [Hosted providers in config.yml](#hosted-providers-in-configyml) for the fields). Config-file entries remain read-only in the dashboard. Stored credentials require `OTARI_SECRET_KEY`, a Fernet key generated with `otari gen-secret-key`. To rotate it, configure the new key before the old key, run the hosted provider and search-tool re-encryption endpoints (`POST /api/v1/hosted-providers/reencrypt` and `POST /api/v1/search-tools/reencrypt`), then remove the old key. Losing every configured encryption key makes stored credentials unrecoverable. #### Upgrading from the Deployment providers page Earlier releases stored runtime providers on a Deployment providers page (`/api/v1/provider-credentials`), which is gone. At startup, each provider it stored becomes a hosted provider of the same name, served by the implementation its `provider_type` names, so an `instance:model` selector, routing policy or alias naming it keeps working. The start asks each provider for its models, so the first start after the upgrade can take a little longer, and offers every model it lists switched on, priced or not, as the stored provider served it, and a later refresh keeps them on, on every path: completions, routing-policy fallbacks, the pass-through endpoints and batches. A model it does not list is refused until it is added on the Hosted providers page, or until "Serve models not on its list" is turned on there. A provider the move cannot take is logged at startup and listed on the Hosted providers page with the reason (`GET /api/v1/hosted-providers/migration`), and serves nothing until it is added again: a provider that lists no models, a name a hosted provider already holds, a key no configured `OTARI_SECRET_KEY` reads, an instance named after another provider (`groq` declared as `openai`), or a keyless Bedrock or SageMaker provider served on the server's own cloud role, which belongs under `providers:` in `config.yml` instead. Two things behave differently once moved. A hosted provider is asked after an organization's own key, where a stored provider was asked before it, so an organization holding its own key is served on that key. And a `config.yml` provider of the same name now serves ahead of it, where a stored provider used to override the configured one. ### Hosted providers in config.yml A hosted provider is a credential the deployment serves requests on when nothing else credentials them (see [Hosted providers](/otari/for-operators/hosted-providers.md#hosted-providers-and-byo-credentials)). Besides the Hosted providers page, a deployment can declare them in a `hosted_providers:` block, keyed by name, with the fields the `/api/v1/hosted-providers` body takes: ```yaml hosted_providers: openai: api_key: ${OPENAI_API_KEY} openai-eu: provider: openai api_key: ${OPENAI_EU_API_KEY} api_base: https://eu.api.openai.com/v1 anthropic: api_key: ${ANTHROPIC_API_KEY} restricted_to_organization_id: 4f1c2a7e-0b9d-4c55-9a0e-2d6f8b3c1e90 vllm: api_base: http://vllm.internal:8000/v1 models: [qwen3-32b] ``` The name is what a request addresses (`openai-eu:gpt-4o`), and what the provider's models, prices and usage are keyed on. It defaults to the implementation; a name of its own lets two endpoints of one implementation be served side by side. A name may not contain `:` or `/`, may not be `otari` or `hosted`, and may not be another implementation's name. | Field | Meaning | | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `provider` | The any-llm implementation serving it. Defaults to the name. | | `api_key` | The key the provider is served on. Required unless the provider needs none (listed below) or has an `api_base` of its own. | | `api_base` | An endpoint other than the provider's default. | | `client_args` | What the provider's SDK client needs beyond the key, such as Bedrock's region. | | `enabled` | `false` keeps the entry without serving it. Defaults to `true`. | | `restricted_to_organization_id` | Serve one organization only, by its id. Checked at boot rather than at load, because the organizations are in the database: an id that names none is logged, and the provider then serves no organization. | | `models` | Model ids a refresh offers when the provider cannot list its own. | | `supported_countries` | The two-letter country codes whose requests it serves. Unset serves every country. See [Supported countries](#supported-countries). | A key may be left out for the self-hosted backends `vllm`, `ollama`, `llamacpp`, `llamafile`, `lmstudio`, `cascadia` and `otari`, for `vertexai` and `vertexaianthropic`, which authenticate through the Google Cloud credentials in the gateway's own environment, and for any provider given an `api_base`, which is called with a placeholder key, as a self-hosted OpenAI-compatible server is. Every other provider, Bedrock included, needs `api_key`. The Hosted providers page and API follow the same rule. As for any hosted provider, a request is served on one only when no organization key or `providers:` instance answers for it first. Reference the key as `${VAR}` rather than writing it into the file: it is resolved from the environment as every `${VAR}` in the file is, so the key stays in your secret store. The whole block can instead be set as JSON through `OTARI_HOSTED_PROVIDERS`, where `${VAR}` resolves the same way: ```bash OTARI_HOSTED_PROVIDERS='{"openai": {"api_key": "${OPENAI_API_KEY}"}}' ``` When it is set, it replaces the `hosted_providers:` block of the file (and of `OTARI_CONFIG_YAML`) whole: a provider only the file names is not served, and the two are never merged per provider. An empty value leaves the file's block in place. The block is validated at load: an unknown implementation, a `provider_type` key (the implementation goes in `provider`), a name a hosted provider cannot take, an unknown field, or a missing `api_key` for a provider that needs one fails the start. With `OTARI_PROVIDER_ALLOW_PRIVATE_HOSTS=false`, an `api_base` that resolves to a private address fails it too, as it is refused for a provider stored through the page; that check runs at boot, because it resolves the host. The block is never shown by `/api/v1/settings`. A configured entry is never written to the database or encrypted. The Hosted providers page lists it with a `config.yml` badge and the API marks it `source: "config"` with no `id`; changing or deleting it through the API answers `409`, so change it in the configuration and restart. Its models, their serving switches and their prices are managed on the page as any hosted provider's are. An entry starts with no models and serves only the ones switched on under it, so refresh its models or add them before it serves a request. A provider stored through the page or the API under the same name wins over the entry, and the start logs each entry it shadows; deleting the stored provider serves the entry again on the same models. Taking an entry out of the configuration leaves its models in the database, serving nothing and offered in no catalog; they come back if the entry does, and the next catalog update run with its cleanup on the Hosted providers page (`POST /api/v1/hosted-providers/catalog-update` with `cleanup`) deletes them and their prices, naming the provider in `orphaned_rosters`. Where no hosted provider offers a model at all, the cleanup deletes those models and leaves every price in place. ### Provider copies A request that asks a provider's own code execution to run over an attached file gets a short-lived copy of that file in the provider's account (see [Files](/otari/for-integrators/files.md#a-file-the-providers-own-code-execution-reads)). The account a copy is in is named by a keyed digest of the credential that made it, so the digest needs a key of its own: `OTARI_PROVIDER_ACCOUNT_PEPPER`. While the pepper is unset, Otari turns `files_provider_upload_enabled` off at startup and logs a warning, so a request that asks a provider's own code execution to run over an attached file is refused. Otari refuses to start when the pepper is shorter than 32 characters or equal to the master key or an `OTARI_SECRET_KEY` key. Generate one with `otari gen-provider-account-pepper` or `openssl rand -base64 32`, and keep it in your secret store. Rotating it costs nothing but a fresh copy of each file the next time a request uses it. ### End-user identifiers Every request a deployment sends to a provider reaches it under one account. A provider that enforces its usage policy per end user needs an identifier for each one, or a block meant for one person lands on the account, and every user and workspace on the deployment loses that provider. So Otari sends each provider call a digest of the workspace and the resolved user, and of the user tag the caller sent Otari when there is one. On chat completions and Responses the tag is `safety_identifier`, or `user` when the request has none; on Messages it is `metadata.user_id`. A tag equal to the resolved user counts as none. The table below names the field each provider receives it in. The provider never sees the tag, which may be personal data, or an Otari id. The same tag from the same user in the same workspace always gets the same identifier, which is what a provider scopes enforcement by. Because the resolved user is always in the digest, a tag only tells apart the end users behind one user: tagging a request with another user's id never gives it that user's identifier. The dashboard's Playground ignores a `user` or `safety_identifier` sent in its request body, so the identifier its calls carry is always the signed-in member's. `user` has a second job: on a service key it names the billed end user, and on any other key it must match the key's user unless `reject_user_mismatch` is off. `safety_identifier` names nobody billed and nothing checks it, so one request can bill one user and tag another. Send it when one key serves several end users that `user` does not tell apart: a request with neither tag gets the key holder's identifier, so a block meant for one lands on all of them. The caller chooses the tag, so a caller that sends a new one on each request gets a new identifier each time and no block lands on anyone. Otari cannot detect that; the answer is to block that customer. | Provider | Field | | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | OpenAI, Azure OpenAI | `safety_identifier` | | Cerebras, DeepSeek, Fireworks, Groq, OpenRouter, xAI | `user` on chat completions (DeepSeek's `user_id`) | | Anthropic, including Azure and Vertex AI | `metadata.user_id` | | Bedrock | `requestMetadata.otari_user`, except `metadata.user_id` on Messages to a Claude model that sets no `output_format` | | Gemini API | the `safety_identifier` label, cut to the 63 characters a label value allows, on chat completions and Messages | | Meta | `safety_identifier` on chat completions | | Any other provider with a native Messages API, on Messages | `metadata.user_id` | Bedrock's abuse detection reads no end-user identifier, but `requestMetadata`, like the request body that carries `metadata.user_id`, lands in the account's invocation logs, which is where the prompt ids in an AWS abuse report are traced back to a user. Every other provider gets nothing: some have no such field (Vertex AI, whose labels are for billing only, and Mistral), some run your own models, and for the rest an unknown field can fail the call. A Responses or Messages request served as a chat completion gets that provider's chat field. The call that describes an image for a text-only model (`vision_describe_model`) carries the request's identifier too, in the field the vision model's provider reads. Nothing turns the identifier off. OpenAI's partner program for gateways requires one on every request, and a provider that cannot take one already gets nothing. Set `OTARI_PROVIDER_USER_IDENTIFIER_KEY` to a random value of at least 32 characters (`openssl rand -base64 32`) to key the digest. Without it the digest is unkeyed, so the same workspace id and user give the same identifier on any deployment, and Otari logs a warning at startup. Otari refuses to start when the key equals the master key or an `OTARI_SECRET_KEY` key. Rotating it gives every user a new identifier: a block a provider placed on an old identifier stops landing, and the provider's history for that user starts again. Rotate it only when the key has leaked. #### Tracing an identifier back A provider that reports abuse names the identifier, and it is a one-way digest. Each usage row stores the identifier its request's provider calls carry, in the indexed `usage_logs.safety_identifier` column, so the database answers who it was: ```sql SELECT workspace_id, user_id, api_key_id, MIN(timestamp) AS first_seen, MAX(timestamp) AS last_seen, COUNT(DISTINCT request_group_id) AS requests FROM usage_logs WHERE safety_identifier = '' GROUP BY workspace_id, user_id, api_key_id; ``` The rows keep an identifier after a key rotation, so an old one still traces. For a Gemini label, which is cut to 63 characters, match the prefix: `WHERE safety_identifier LIKE '