Otari routes requests to LLM providers through any-llm-sdk. This page covers the model format, supported providers, and capabilities.
Model format
Models are specified as provider:model_name:
openai:gpt-4o
anthropic:claude-sonnet-4-6
mistral:mistral-large-latest
vertexai:gemini-2.0-flash
The provider prefix tells Otari which backend to route to. The model_name is passed directly to that provider's API.
Supported providers
Otari depends on any-llm-sdk[all]. Provider support can change as the SDK evolves.
Use this list as a quick reference for common providers supported by the current Otari build.
Provider | Config key | Example model | Notes |
Anthropic |
|
|
|
AWS Bedrock |
|
| AWS credentials required |
Azure OpenAI |
|
| Requires |
Azure Anthropic |
|
| Requires |
Cerebras |
|
|
|
Cohere |
|
| Also supports rerank |
DashScope |
|
| Alibaba Cloud |
Databricks |
|
| Requires |
DeepInfra |
|
|
|
DeepSeek |
|
|
|
Fireworks |
|
|
|
Gemini |
|
|
|
Groq |
|
|
|
HuggingFace |
|
| Pin a backend with |
Inception |
|
|
|
Llama.cpp |
|
| Local server |
Llamafile |
|
| Local server |
LM Studio |
|
| Local server |
MiniMax |
|
|
|
Mistral |
|
|
|
Moonshot |
|
|
|
Nebius |
|
|
|
Ollama |
|
| Local server |
OpenAI |
|
|
|
OpenRouter |
|
|
|
Perplexity |
|
|
|
SageMaker |
|
| AWS credentials required |
SambaNova |
|
|
|
Together |
|
|
|
Vertex AI |
|
| Requires service account |
Vertex AI Anthropic |
|
| Requires service account |
vLLM |
|
| Self-hosted |
Voyage |
|
| Embeddings only |
WatsonX |
|
|
|
xAI |
|
|
|
Capabilities
Not all providers support all endpoints. Here's what each endpoint type requires:
Endpoint | Capability | Example providers |
| Chat completion | Most providers |
| Anthropic Messages API | Anthropic, Vertex AI Anthropic |
| OpenAI Responses API | OpenAI |
| Text embeddings | OpenAI, Cohere, Voyage, Vertex AI |
| Content moderation | OpenAI |
| Document reranking | Cohere |
| Image generation | OpenAI, Vertex AI |
| Audio transcription | OpenAI |
| Text-to-speech | OpenAI |
| Batch processing | OpenAI, Anthropic |
In deployments connected to otari.ai, the final model/provider choices are resolved by otari.ai routing policy, not by local providers configuration.
Configuring a provider
In config.yml:
providers:
openai:
api_key: "sk-..."
api_base: "https://api.openai.com/v1" # optional for hosted OpenAI
Or via environment variable:
export OPENAI_API_KEY="sk-..."
Both approaches work. Config file values take precedence over environment variables.
In standalone mode, provider config only tells Otari how to reach the backend. Otari also requires pricing for the model you call by default, unless default_pricing covers it or require_pricing: false is set.
For the full configuration reference, see Configuration.
Named provider instances
The providers map is keyed by instance name. Most of the time that name is also the provider, such as openai or anthropic.
If you want to use multiple backends that share one provider implementation, give one of them a custom name and set provider_type. This is common for self-hosted OpenAI-compatible servers such as vLLM, llama.cpp, or LM Studio:
providers:
openai: # key is a real provider, no provider_type needed
api_key: ${OPENAI_API_KEY}
home_lab: # custom instance name
provider_type: openai # underlying any-llm implementation
api_base: "https://nathans-mac-studio.example.ts.net/v1"
api_key: ${HOME_LAB_TOKEN}
Route to a named instance with instance_name:model. For example, home_lab:deepseek-v4-flash uses the home_lab config, but Otari sends the request through the OpenAI provider implementation with that instance's api_base and api_key. openai:gpt-4o still uses the regular openai config.
Pricing and usage are keyed on the instance name, so configure pricing under home_lab:deepseek-v4-flash if you want that model to be priceable. If you do not set provider_type, the key works as before and names the provider directly.
provider_type: openai-compatible and provider_type: openai_compatible are both accepted as aliases for openai.
Named instances are a standalone-mode feature. In hybrid mode, provider credentials come from otari.ai per request, so local named instances are not used.
Declaring models for backends without /v1/models
/v1/models lists an instance's models by calling the backend's model-listing endpoint. When a backend does not expose /v1/models, declare the served model ids so they still appear in the listing:
providers: edge_box: provider_type: openai api_base: "https://edge.example.ts.net/v1" api_key: ${EDGE_TOKEN} models: - llama-3.3-70b - qwen3-32bThe declared models are listed as edge_box:<model>. Direct requests work with or without this list; it only affects discovery.
Listing available models
Query Otari to see which models are available:
curl http://localhost:8000/v1/models \ -H "Authorization: Bearer <your-api-key>"
Provider-specific notes
Pinning a HuggingFace inference backend
HuggingFace Inference Providers is a router: the same model id (for example zai-org/GLM-4.6) can be served by several backends (Together, Novita, and others), and in the default "auto" mode the backend, and therefore the price, is chosen at request time. To route (and price) deterministically, pin a backend with a :<backend> suffix on the model id, which the HuggingFace router honors server side:
huggingface:zai-org/GLM-4.6:together huggingface:zai-org/GLM-4.6:novita
The pinned-selector grammar is huggingface:<model>:<backend>. Otari splits the provider off the first :, so everything after it (<model>:<backend>) is forwarded as the model id and the :<backend> suffix reaches the router unchanged. The router also accepts policy suffixes such as :cheapest, :fastest, :preferred, and :auto.
This grammar is the contract consumers build against. The otari.ai platform's pricing UI, for instance, offers each priceable backend as a pinned huggingface:<model>:<backend> selector, because a pinned selector resolves to a single backend, which is what makes a HuggingFace model priceable (auto mode cannot be priced from the model id alone).