> ## Documentation Index
> Fetch the complete documentation index at: /docs/llms.txt
> Use this file to discover all available pages before exploring further.
# LLM traces for debugging
> Explore step-by-step details of every LLM generation. Debug RAG pipelines, evaluators, guardrails, and caching with full workflow visibility and cost tracking.
## What are Traces
**Traces** record every AI request handled by **Toqan Router**, from a single model call to a multi-step **Agent** run. Each trace shows the full workflow of one request as a tree of steps (spans), with inputs, outputs, latency, token usage and cost for each step.
Traces are created for:
* Model calls sent through the [AI Gateway](/docs/ai-gateway/using-the-router.md)
* Agent runs, including sub-agent and tool calls
* Deployment invocations
* Coding sessions from code assistants connected to **Toqan Router**
* Application code instrumented with [OpenTelemetry frameworks](/docs/ai-studio/integrations/frameworks/overview.md)
A trace contains a step for each operation in the request, such as:
* LLM calls, including Retries and Fallbacks
* Tool and sub-agent calls
* Evaluators and Guardrails
* Retrieval from a Knowledge Base, including Embeddings
* Caching
## Creating Traces
Traces are generated automatically for requests through the **AI Gateway**, **Agents** and **Deployments**. Create custom traces for application code using framework instrumentation or the OpenTelemetry API.
### Framework Instrumentation
For popular AI frameworks and libraries, use automatic instrumentation with OpenTelemetry:
> [card] [Framework Integrations](/docs/ai-studio/integrations/frameworks/overview.md)
>
> Explore automatic instrumentation for OpenAI, LangChain, LlamaIndex, CrewAI, Autogen, and 15+ other frameworks with OpenTelemetry integration.
### Custom application spans
Use the standard OpenTelemetry API for custom application spans. Configure the exporter with the regional endpoint from the [Observability quickstart](/docs/ai-studio/observability/quickstart.md), then wrap work in a span:
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from opentelemetry import trace
tracer = trace.get_tracer("document-pipeline")
with tracer.start_as_current_span("process_document") as span:
span.set_attribute("document.id", "doc-123")
# Run the application's document processing here.
```
Child spans created inside this context keep the parent-child hierarchy. Avoid recording secrets or personal data in span attributes.
## How to Lookup Traces
To find traces, open **Traces** in the **Observability** section of the sidebar.
The page has three parts:
* **Query bar**: search, filters, time range, and the Views, Fields, Review and Reload buttons.
* **Activity chart**: the number of **Successful** and **Errors** traces over the selected time range. Click a bar, or drag across several, to zoom into that range; the back button restores the previous range.
* **Traces table**: one row per trace, newest first. Scroll down to load more traces.
The page header shows **Total cost** for the traces that match the current time range, filters, search or OQL query and project. It sums the whole matching set, including [evaluator](/docs/ai-studio/observability/trace-evaluations.md) cost, not only the rows loaded in the table.
Select a row to open the trace in a side panel, showing the **hierarchy** of events in order of execution and the details of the selected step. With a trace open, press J and K to move to the next or previous trace.
The side panel stacks collapsible sections for the selected step:
| Panel | Shows |
| - | - |
| **Properties** | **Trace ID**, **Span ID**, **Created**, **Message count**, **Model**, **Input Tokens**, **Output Tokens**, **Total Tokens**, and **Estimated Cost**. |
| **Input** and **Output** | The step's payload, as a collapsible tree. |
| **Annotations** | The annotations attached to the step. |
| **Metadata** | The step's metadata and attributes. |
| **Economics** | **Input tokens**, **Output tokens**, **Total tokens**, **Total cost**, **Input cost**, and **Output cost**. |
A step that stopped before finishing carries a **Stopped early** warning naming the cause: output token limit reached, response blocked by a content filter, an agent limit reached, or an abnormal end. The raw value also appears as **Finish reason** in **Properties**.
Variables passed to an agent in the `variables` field are recorded as the `orq.variables` span attribute, so they appear under **Metadata** in the side panel. See [Span Attributes](/docs/ai-studio/observability/span-attributes.md) for the field list.
**Add to dataset** copies the selected step into a dataset, and **Add to annotation queue** sends it to an annotation queue. When the step carries annotations, **Copy annotations** adds each one as a named input; the checkbox is unavailable when an input named `annotations` already exists.
Turn on **Live** to refresh the list every 10 seconds. Live pauses while a trace is open.
> [!TIP]
>
> Search, filters, columns and time range are stored in the page URL. Copy the URL to share the exact list with a teammate.
### Trace Views
Each trace can be inspected in three views to suit different debugging and analysis needs.
> [tab] Trace
>
> The **Trace** view shows the full execution tree for a single run. Each step is displayed hierarchically, including LLM calls, tool invocations, knowledge retrievals, and memory interactions. Use this view to inspect inputs, outputs, token usage, and latency at every step.
>
> For multi-agent runs, the hierarchy renders as an **agent graph**: parent agents and their sub-agent calls are shown as a nested tree, making it easy to follow how work was delegated across agents and where time was spent. See [Agent Graphs](/docs/ai-studio/observability/agent-graphs.md) for how graphs are produced and how to debug delegation paths.
>
> > [!TIP]
> >
> > Erroring Traces are shown with a icon.
>
> > [!NOTE]
> >
> > Manually evaluate responses using **Annotations** directly on individual spans. Annotations defined in the project are available on all spans automatically. Learn more in [Annotations](/docs/ai-studio/observability/annotations.md).
> [tab] Thread
>
> The **Thread** view presents the execution as a conversation thread, showing the sequence of messages exchanged between the user, the agent, and any tools. Use this view to follow the narrative flow of a task and understand how the agent reasoned through a problem.
> [tab] Timeline
>
> The **Timeline** view shows execution steps plotted against time, making it easy to identify bottlenecks, measure step durations, and understand which operations ran sequentially or in parallel.
### Search and Time Range
The search box matches part of a trace ID, thread ID or session ID, and the name, span ID, model, provider, agent name or tool name of any span in the trace. Matching is case-insensitive.
Select the time range picker to choose a preset from **5 mins** to **90 days**, or set a **Start date** and **End date** for a custom range. Presets longer than the workspace's data retention are disabled. The last preset used is remembered for the next visit.
> [!TIP]
>
> Switch the query bar from **Builder** to **OQL** to write the query as a pipeline, for example `fetch traces | filter status == "error" | sort end_time desc | limit 100`. See [Trace and log selection with OQL](/docs/ai-studio/observability/oql.md).
### Viewing Errors
Traces that encountered an error show a red **error** status badge in the list, and are counted as **Errors** in the activity chart. Add a **Status** filter with the value `error` to scope the list to errors only. The **success** badge covers traces that the Telemetry API reports as `ok` or `unset`; see [Trace status values](/docs/ai-studio/observability/telemetry-api.md#trace-status-values).
The **Status** filter combines with any other filter, for example:
* Filter by **Status** `error` and an [**Identity**](/docs/ai-studio/observability/identities.md) to see all errors for a specific user.
* Combine with **Project** or a metadata field to narrow down failures to a particular environment or deployment.
Save the filter as a [view](#creating-custom-views) to return to it in one click.
### Filtering Traces
1. Click Filter in the query bar.
2. Select a field. Search the list to find it faster. Common fields such as **Status**, **Model** and **Trace name** are listed first, followed by the attributes and metadata fields found in the workspace's traces.
3. Select an operator, for example `==`, `!=`, `contains`, `in`, `between` or `is empty`.
4. Pick a suggested value or type one, then click Filter.
The first filter applies immediately. Each filter then appears as a chip below the query bar:
* Click a chip to change its operator or value, or click to remove it.
* Click to add another filter.
* With two or more filters, choose **and** to match all of them or **or** to match any of them.
* Changes to the filters wait until **Run** is clicked. **Clear** removes all filters at once.
> [!NOTE]
>
> When working with Agents, access traces directly from the Agent page with automatic filtering for that specific agent.
### Filter grammar
The **Search traces** and **Search logs** APIs take a `filters` array alongside the free-text `query`. Each entry is one condition:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"field": "status",
"op": "in",
"values": ["error"]
}
```
| Key | Value |
| - | - |
| `field` | Field name, up to 128 characters of letters, digits, and `_ . : -` |
| `op` | Comparison operator, from the set below |
| `values` | String array, up to 100 entries of up to 4096 characters each. Numeric and timestamp values are sent as strings |
`filter_operator` combines the entries with `and` (default) or `or`. A request accepts up to 20 filters. `from` and `to` are required and bound the search window; `sort`, `limit`, and `page_token` follow the [Search traces reference](/docs/reference/traces/search-traces.md) for traces and the [Search logs reference](/docs/reference/logs/search-logs.md) for logs.
The `op` values are `eq`, `neq`, `in`, `not_in`, `gt`, `gte`, `lt`, `lte`, `between`, `contains`, `exists`, and `not_exists`. The table lists the operators each type commonly accepts; a specific field may accept fewer or more:
| Field type | Operators | Example |
| - | - | - |
| String | `eq`, `neq`, `in`, `not_in`, `contains`, `exists`, `not_exists` | `{"field": "name", "op": "contains", "values": ["checkout"]}` |
| Number | `eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `between`, `exists` | `{"field": "tokens.total", "op": "gt", "values": ["1000"]}` |
| Timestamp | `eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `between`, `exists`, `not_exists` | `{"field": "start_time", "op": "gte", "values": ["2026-10-01T00:00:00Z"]}` |
| Boolean | `eq`, `neq`, `exists`, `not_exists` | `{"field": "eval_passed", "op": "eq", "values": ["true"]}` |
| Enum | `eq`, `neq`, `in`, `not_in` | `{"field": "status", "op": "in", "values": ["error"]}` |
`between` takes exactly two values, the lower bound first. `exists` and `not_exists` take no values.
Field-level support varies, so call `GET /v3/traces/fields` for the filter fields, their types, and the operators each one accepts. Log search uses the same filter object and operator vocabulary.
**Filter one identity's errors.** Set `filter_operator` to `and` and combine the conditions in a single request:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"from": "2026-10-01T00:00:00Z",
"to": "2026-10-02T00:00:00Z",
"filter_operator": "and",
"filters": [
{ "field": "identity_id", "op": "eq", "values": [""] },
{ "field": "status", "op": "in", "values": ["error"] }
]
}
```
`identity_id` holds the identity a request was attributed to: the value passed as `identity.id` on an Agent response or Deployment invocation, or in the `X-TOQAN-IDENTITY-ID` header. It matches the **Identity** filter in the **AI Studio**.
**Filter by API key.** Every span records the key that made the request, as its identifier rather than the secret, in `orq.api_key_id` (see [Span attributes](/docs/ai-studio/observability/span-attributes.md#tenant-scope)). Filter with the **API Key ID** field in the query bar, or pass the condition to `Search traces`:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{ "field": "api_key_id", "op": "eq", "values": [""] }
```
For spend per key, break down [Activity](/docs/ai-studio/observability/activity.md#spend) by **API keys**.
### Configuring Columns
Click Fields to open the **Fields** panel and choose which columns the table shows. Every attribute and metadata field found in the workspace's traces can be shown as a column, next to the built-in columns: **Start time**, **Status**, **Name**, **Duration**, **Total tokens**, **Input tokens**, **Output tokens**, **Cached input tokens**, **Total cost**, **Project**, **Metadata**, **Signals**, and **Trace ID**.
* **Show or hide**: check or uncheck a field. Search the panel to find a field by name.
* **Reorder**: switch to **Selected** and drag a field by to move its column. Reordering only works while all selected fields are visible, so clear the search box first.
* **Resize**: drag the right edge of a column header.
* **Rename**: open on an attribute or metadata column header and select **Edit name** to give it a display name.
* **Remove**: open on a column header and select **Remove** to hide it.
Column widths and order are remembered in the browser. To keep a column layout and share it with the workspace, save it in a view.
### Creating Custom Views
A view saves the current filters, columns and column names under a name, to reopen them later from the Views dropdown.
To create a view:
1. Set the filters and columns.
2. Open Views and select **Create new view**.
3. Enter a **Name**.
4. Optionally check **Make this view private for myself**. Views are shared with workspace members by default.
5. Click **Create View**.
To update a view, select it, change the filters or columns, then open **Save** next to the filter chips and select **Save to this view**. When only the columns changed, click **Update view** in the query bar instead.
Hover a view in the dropdown to manage it:
* : set or unset it as the default view. The default view opens automatically on the Traces page.
* \> **Edit**: change its name or privacy.
* \> **Share**: copy a link that opens the view with its filters.
* \> **Delete**: delete the view.
Select **Default view** to return to the standard columns with no saved filters.
> [!NOTE]
>
> Views saved on the previous Traces page were moved to the new page. Filters that the new page cannot express were adjusted or removed.
Manage saved Views programmatically through the [Views API](/docs/ai-studio/observability/views-api.md).
## Review Traces
Click Review in the toolbar to open the review screen and step through traces one by one. Applying one or more [filters](#filtering-traces) first narrows the review queue to matching traces; with no filter active, Review steps through the current list.
The screen follows the same three-panel layout as [Annotation Queues](/docs/ai-studio/observability/annotation-queues.md#use-annotation-queues):
* **Left**: Inputs and Metrics for the selected span.
* **Center**: Full trace conversation.
* **Right**: [Annotation](/docs/ai-studio/observability/annotations.md) controls for manual review. Select in the Annotations panel header to [create a new annotation](/docs/ai-studio/observability/annotations.md#create-annotations) without leaving the screen.
Annotations applied here are written back to the span and are queryable via the **Toqan Router MCP**.
## Correct an Evaluator Result
Hover an Evaluator result to reveal a , and select it to open a popover with:
* **Value**: the corrected result, matching the Evaluator's own output type (a toggle for boolean Evaluators, a text or number field otherwise).
* **Explanation**: an optional comment.
Select **Correct** to submit, or **Update** to replace an existing correction. The correction appears under the original result as **Corrected to \**, along with the explanation and the reviewer's name.
Corrections are written back to the Trace like any other annotation. This gives a basis to compare an Evaluator's own results against reviewer corrections over time, using that signal (for example through the **Toqan Router MCP**) to see where an Evaluator's configuration may need adjustment.
> [!NOTE]
>
> This corrects the Evaluator's *result*, not the AI response text. For rewriting the response itself, use the Logs [**Add a Text Correction**](/docs/ai-studio/observability/annotations.md#use-annotations) flow.
> [!NOTE]
>
> See [Correct an Evaluator Result](/docs/ai-studio/observability/annotations.md#use-annotations) in the API & SDK tab for the API/SDK equivalent.
## Threads
> [card] [Threads](/docs/ai-studio/observability/threads.md)
>
> Visualize conversation history as Threads to follow the full sequence of messages across an agent session.
## Reference
> [card] [Token and Cost Tracking](/docs/ai-studio/observability/token-cost-tracking.md)
>
> How token usage is captured and cost is calculated per LLM request.
> [card] [Span Attributes](/docs/ai-studio/observability/span-attributes.md)
>
> Complete reference for all `orq.*` span attributes emitted on traces, webhook payloads, and trace exports.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.