> For the complete documentation index, see [llms.txt](https://docs.otari.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.otari.ai/otari/for-integrators/traces.md).

# Agent traces

Otari records an **agent trace** for every completion it serves: the whole session an agent ran, the turns inside it, and inside each turn every request, LLM call, routing attempt, guardrail check, MCP connection and tool call, with their timing and cost. An LLM call is one kind of span among several, so a trace answers what an agent run did and cost, not only what one call did.

No prompt, model output or tool content is recorded. A trace holds names, ids, timings, token counts, sizes, outcomes and costs. The sizes are numbers only, described under [The context map](#the-context-map).

An LLM call's input tokens are its whole prompt, cached tokens included, whichever way its provider reports them. Beside them the call keeps how many input tokens were read from or written to the prompt cache, and how many output tokens went to reasoning.

## The context map

A session's **At a glance**, shown before a span is picked, maps what fills the model's context window, as blocks sized by tokens, with the unused part of the window as **Free**. It is read from the session's last LLM call that recorded input, since each call carries the whole conversation so far, and it also names the fullest call when that was an earlier one. Hover, focus or tap a block for its tokens and share of the window. The window comes from the model catalog when the model is listed there. Without it the map shows no free space and says the window is unknown. The total is always the provider's exact input-token count. How it is split depends on what the call recorded:

* **Every call**, including those a coding agent exports over OTLP, is split into cache reads, cache writes and fresh input, from the cache counts the call already keeps.
* **A request Otari served itself** (chat completions, Messages or Responses) is also split into the system prompt, tool definitions, files, messages and tool results. Otari has no tokenizer, so it counts characters per part as the request arrives, and the dashboard scales those weights so they add up to the exact input tokens. The split is an estimate and the map says so. An image, audio clip, file or document counts for a flat allowance of 4,000 characters (about 1,000 tokens), because those are billed by size and format rather than by bytes. Tokens the provider adds itself, such as a hidden tool prompt, are spread across the parts.

For that estimate a span keeps eight whole numbers, never text: the characters in the system prompt, the tool definitions, the user messages, the assistant messages, the tool results and the files, and the number of tool definitions and of messages. They are stored as the span attributes `otari.context.system_chars`, `otari.context.tools_chars`, `otari.context.user_chars`, `otari.context.assistant_chars`, `otari.context.tool_result_chars`, `otari.context.file_chars`, `otari.context.tool_count` and `otari.context.message_count`. Every attempt of a request, a fallback included, carries the same numbers, and turning trace capture off stores none. A request that runs a tool loop inside the gateway reports the sizes of the request as it arrived, not of each later round.

## What makes a session

Requests are grouped into one session only when the client names the session. Otari never guesses that two requests belong together from their content.

| Signal                             | Example                                                                                                               | Shown as  |
| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------- | --------- |
| The `Otari-Conversation-Id` header | `Otari-Conversation-Id: chat-42`                                                                                      | `client`  |
| The `session_label` request field  | `"session_label": "chat-42"`                                                                                          | `client`  |
| A `session_id` request tag         | `"metadata": {"session_id": "chat-42"}`                                                                               | `client`  |
| The agent harness's own session id | Claude Code's `x-claude-code-session-id`, sent on every request                                                       | `harness` |
| The agent harness's own session id | Codex's `thread-id`, or `session-id` without one, on a request whose `originator` or `User-Agent` starts with `codex` | `harness` |
| An OTLP span's Claude session      | `anthropic.session.id` on an exported span                                                                            | `otlp`    |
| An OTLP span's conversation        | `gen_ai.conversation.id` on an exported span                                                                          | `otlp`    |
| A Claude Code span's session       | `session.id` on a span Claude Code exported                                                                           | `otlp`    |

A request that names no session is a trace of its own. To group a custom client's requests, send one of the signals above.

A session belongs to the user whose API key sent it: two people who choose the same label get two sessions, and a session id never reaches another workspace.

A gateway request's usage row records its session's trace id as `source_label`, so Usage's spend by session, and `by_source_label` in `GET /api/v1/usage/summary`, group the same sessions this page lists, whether or not traces are captured. The id is the one a session's entry shows, so a row in one finds its match in the other. A request that names no session reads as "(no session)" there. Usage imported from outside the gateway keeps the label it was sent with.

## Sessions from your own agent

Claude Code and Codex are recognized on their own. An agent or harness of your own needs two things on its requests to read well as a trace.

**Send one session id on every request of a run.** The `Otari-Conversation-Id` header is the one to use: it works the same on every completion endpoint, and it leaves `session_label` free for whatever you report spend by. Send it on each request of the run, including the follow-ups that hand tool results back and any retries, since a request without it starts a trace of its own.

* When more than one signal is present, the header wins, then `session_label`, then the `session_id` tag, then a harness's own id. A label you set per customer or per automation for spend therefore groups every run under it into one session, unless each run also sends its own header.
* The value is any string up to 512 characters. Otari hashes it into the trace id, so the id differs per workspace and user. When the value is identifier-shaped (up to 128 letters, digits and `_ . : / @ + -`, and not shaped like an email address, a file path or a credential), it is also stored as the session's name: the Sessions list shows it, and its search box finds the session by any part of it, in any case (`hackbot-1234`, `hackbot-` or `BOT-12`). The same box also matches any part of a session's id, agent or user. Any other value is not stored. A harness's own id and an OTLP session id are never stored as a name.
* Give each run its own id rather than reusing one for days: after `trace_session_max_age_days` a reused id starts a new trace anyway.
* Give a subagent an id of its own. Under its parent's id, the task it was handed reads as a new turn of the parent's session.

**Name the harness in `User-Agent`.** The first product token is shown as the session's harness: `my-agent/1.4.0` shows `my-agent`. An SDK's default (`OpenAI/Python 1.x`, `Anthropic/Python 0.x`) shows the SDK's name instead.

Turns and the tools your agent runs are read from each request's input, so they need nothing extra when the request carries the conversation as usual: the user's message last when a turn starts, the tool results last when the model's tool calls are answered, each with the call id the model gave it. In a Messages request, any `system` messages after them are skipped; Claude Code sends one on almost every request. A Claude Code subagent sends its parent's session id and an `x-claude-code-agent-id` of its own, and its requests open no turn, since it works inside its parent's turn. A Responses request that continues with `previous_response_id` carries only the new items, so its tools are recorded without a name.

## Turns, steps and tools

* A **step** is one request to the gateway, named for the endpoint that served it, such as `/v1/chat/completions`. Every retry is a step of its own, with its own cost.
* A **turn** starts at a step whose last input is the user's own text, and includes the steps after it that hand tool results back to the model. A turn is **failed** when its last step failed, **incomplete** when the response was cut off, **active** while the session has been busy in the last ten minutes, and **completed** otherwise. A routing attempt that failed and was recovered by a fallback does not fail its turn.
* **Tools the gateway runs** (web search, web fetch, code execution, MCP tools) are timed as they run.
* **Tools the client runs** (a shell, a file read in a coding agent) are recorded from the request that carries their result: the tool's name, whether it returned an error, and when the result arrived. Their start is when the step before ended, so their duration is marked approximate.

## Sessions and failures

A session's error count is every span in it that ended in an error a fallback did not recover: a failed request, a failed provider call, a guardrail failure, a failed connection to an MCP server, and a tool that returned an error. The session series (`GET /api/v1/traces/series`, the Sessions chart) counts a session as **failed** only when one of those errors is not a tool's. A tool's error, such as a missing file or a search that found nothing, goes back to the model, which usually carries on, so a run whose requests all succeeded counts as succeeded however many tool errors it recovered from. That includes an MCP tool: a call the server answers with `isError`, or one that fails in transport, reaches the model as a tool error too. Only a failed connection to an MCP server fails the session. The error count, and the `has_error` filter, still include every one of them.

## Instrumented agents

An agent that exports OpenTelemetry spans to `/otlp/v1/traces` (see [Importing external usage](/otari/for-integrators/external-usage.md)) has every span kept on its trace, including its own `invoke_agent` and `execute_tool` spans, with the span tree it sent. Span names and attribute values are kept only when they are identifier-shaped. A span's `gen_ai.response.finish_reasons` array is kept as one value, its reasons joined by a comma: `stop,length`. The read API returns each of these spans with the `otel_trace_id` and `otel_span_id` it was exported with, in lowercase hex, so it can be joined back to the same trace in another backend. A span the gateway records itself has neither.

### Claude Code without routing

Claude Code can export its own spans, so a session is traced without sending its model traffic through Otari. [Send Claude Code sessions to Otari](/otari/for-integrators/use-with-claude-code.md#send-claude-code-sessions-to-otari) has the whole setup, guardrail checks included. In short, with a budget-exempt API key:

```bash
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1
export OTEL_TRACES_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otari.example.com/otlp"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer tk-your-import-key"
claude
```

Span export is a Claude Code beta. Each prompt becomes a turn, with its model calls and the tools it ran. A tool call keeps a digest of its id, never the id itself, and records who allowed it or refused it: a tool a hook refused, such as an [agent guardrail](/otari/for-integrators/agent-guardrails.md), is `blocked` with the decision source `hook`. A blocked tool call is a guardrail doing its job, not an error, and does not count toward a session's errors. Nor does a tool call that returned an error, such as a command that exited non-zero: the agent reads that error like any other result, and `grep` finding nothing or a test failing on purpose returns one too. A tool the gateway runs itself (an MCP tool, web search, web fetch or code execution) still records an error result as an error, as [Sessions and failures](#sessions-and-failures) describes. A short `claude -p` run can exit before its last spans are exported.

A session's cost comes from its logs, not its spans: Otari prices each model request's `api_request` event at the deployment's rates, cache reads and writes included, and shows the cost on the request and its session. Without `OTEL_LOGS_EXPORTER=otlp` a session has its tokens but no cost. The cost is what the requests are worth at API prices, so on a Claude subscription it is not what was charged.

A session's header counts its tool calls and the gates its hooks ran, each broken down on hover: the calls by tool, and the gates by what they decided. **At a glance** opens a session with where its time and money went: how long it took, with its turns and tool calls, and what it cost, with its requests and tokens, each above one bar and a figure per part. Below them, what fills the model's context window (see [The context map](#the-context-map)), then the gates its hooks ran and what they decided.

* **Time** splits the session into model requests, hooks, running tools, waiting for you to approve a call, and idle between turns. When idle is more than half the session, the bar draws it shorter, marked `//`, and its figure keeps the true time. Running tools opens onto each tool's calls and running time. A call that started a subagent runs until the subagent's last span, so its time also counts as model time, and the panel says when parts overlap like that. Tools that ran inside the request that called them, as a gateway's built-in web search and code execution do, get a lane of their own under the model's part. Each gate is timed by its own run, so a slow judge gate shows on its own.
* **Cost** is the session's total, split by meter (fresh input, output, cache reads and cache writes) when the requests were priced by the token, with what reading from the cache saved over paying for those tokens as fresh input. Requests priced whole, as a gateway prices a routed request, have no split. A session with nothing priced says so rather than $0.00.

A judge gate that [reports its model calls](/otari/for-integrators/agent-guardrails.md#reporting-to-otari) is priced like a request and counted in the session's cost, though not in any turn's. Its check shows the model, the tokens and the cost, and **At a glance** says what the judge gates cost in all.

A turn shows what its requests cost. A tool call shows an estimate, marked `≈`, because a provider bills requests, not tool calls. It is read off the requests either side of the call, in the context the call ran in (a subagent has its own):

* **Calling it:** its share of the output of the request that called it.
* **Adding its result:** how much the next request's context grew, at what that request paid for new input, usually a cache write.
* **Re-reading it:** the result again on every later request, at the cache-read rate, or the cache-write rate once the cache expired, until the context is compacted.

Calls made together share these equally, since a request reports how much its context grew, not how much of that each result was. An `Agent` call also carries what its subagent's requests cost, which is exact. The estimate assumes one main model per context, so in a session that switches models with `/model` partway through, the calls around the switch get rougher estimates.

**Cost by tool** adds the estimates up for the session: each tool's calls and cost, and its share of the session. A subagent's calls are listed under the call that ran it, with the models the subagent used and what its own reasoning and answers cost, and are counted there once. What judge gates cost is listed apart. What no tool call accounts for, such as the prompts and the answers, is listed last. It is where to look before moving a tool's work, or a subagent, to a smaller model.

### Keeping what a span was about

A trace keeps no prompt, command or file path. A deployment can keep them by setting `OTARI_TRACE_CONTENT_ENABLED=true`. They are stored encrypted with `OTARI_SECRET_KEY`, the key that encrypts stored provider credentials, and are not kept at all on a deployment without it. They are kept only when the agent sends them:

* Claude Code sends the prompt that opened a turn with `OTEL_LOG_USER_PROMPTS=1`, and each tool's command or file path with `OTEL_LOG_TOOL_DETAILS=1`. Otari never keeps a tool's output, even when Claude Code exports it (`OTEL_LOG_TOOL_CONTENT=1`).
* `otari hook` sends the command or paths a gate checked, and the gate's own detail, with `OTARI_HOOK_REPORT_CONTENT=1`.

A span shows what it was about as its input, and a tool call is named for its command or file in the tree. A span's content is sealed for that span alone, and reads back empty once no configured key opens it, so keep a rotated-out key in `OTARI_SECRET_KEY` for as long as you keep traces. Only the deployment operator, an owner or admin of the organization, and the user a session belongs to read its content; any other member reads the session with every span's content empty.

## Reading traces

| Endpoint                                                                           | Who                                                                                                                                 |
| ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `GET /api/v1/traces`, `/count`, `/series`, `/{trace_id}`, `/requests/{request_id}` | the deployment operator, every workspace of their active organization                                                               |
| `GET /api/v1/organizations/me/traces`, and the same four                           | a signed-in member: an owner or admin reads every workspace of their active organization, anyone else the workspaces they belong to |

`/series` counts the sessions active in each bucket of a window: a session is active from its start to its last activity, so a long one counts in every bucket it ran through. `bucket` is `1m`, `5m`, `30m`, `hour`, `6h` or `day`. Every read takes the window the same way: `start` and `end` keep the sessions active between them (last active at or after `start`, started before `end`), so a session still running past `end` is listed in the window it was counted in.

A list read narrows to one workspace with `workspace_id`, inside the caller's scope and never beyond it. `q` matches the start of a session's trace id or of its name, case-sensitively, inside the same scope. A trace is keyed by its workspace and its id, so a detail read takes the `workspace_id` its list entry names; without it, an id in more than one of the caller's workspaces answers 409 rather than a guess. `/requests/{request_id}` answers the session a gateway request was traced into, by the `Otari-Request-ID` it was sent (a usage row's `request_group_id`), and 404 where the scope holds no trace of it. A trace outside the caller's scope answers 404.

Deleting a user erases their traces, as it erases their telemetry.

## Settings

| Setting                      | Default |                                                                                                                          |
| ---------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------ |
| `trace_capture_enabled`      | `true`  | Record traces and serve the read API. Requires restart.                                                                  |
| `trace_retention_days`       | `30`    | Delete a trace once it has had no activity for this long.                                                                |
| `trace_session_max_age_days` | `90`    | Delete a session's trace this long after it started, even while it is active, so a reused session id starts a new trace. |

Traces are written off the request path, within fixed limits: a trace never slows or fails a request, and when the writer cannot keep up it drops whole requests' spans rather than parts of them. One OTLP export adds at most 1,000 spans in at most 100 sessions. The `gateway_trace_spans_dropped` metric counts what was dropped, by reason.

The cost on a trace is a snapshot of the usage rows' cost. Usage stays the billing record.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.otari.ai/otari/for-integrators/traces.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.