> ## Documentation Index
> Fetch the complete documentation index at: /docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Trace evaluations

> Attach evaluators to Deployments and Agents in Toqan Router to score sampled production traces asynchronously for continuous quality monitoring.

Trace evaluations run Evaluators on production traffic passing through a Deployment or Agent. Results appear as spans in the [Traces](/docs/ai-studio/observability/traces.md) hierarchy, providing continuous quality monitoring without manual review.

## How trace evaluations work

An Evaluator attached to a Deployment or Agent runs on sampled requests. Input Evaluators score the prompt before the model generates; Output Evaluators score the response after generation. Both run **asynchronously** and never block the caller.

This differs from Guardrails, which run **synchronously** and can deny a generation if the evaluation fails.

> [!NOTE]
>
> Evaluators do not run when using the Test panel in AI Studio. Invoke the Deployment via the API or SDK to trigger evaluators.

## Trace scrubbing and evaluator data

[Trace Scrubbing](/docs/ai-gateway/configuration/routing-rules.md#plugins) changes the data stored in traces. It does not change the data processed by an **Evaluator** or **Guardrail**. The check still receives the original runtime input, output, system instructions, and variables before its spans are scrubbed for storage.

When the originating call scrubs any content category (`system`, `input`, `output`, or `variables`), the policy applies to the complete evaluator subtree. All content categories are scrubbed together because an evaluator prompt, explanation, or Python `stdout` can repeat data from any of them. Selecting only `metadata` does not trigger evaluator content scrubbing.

| Trace data | Stored result |
| - | - |
| Evaluator input snapshot | Stored as an empty object. |
| LLM Evaluator child messages | Preserve message structure, roles, and tool-call names, but store empty text, reasoning, and tool arguments. |
| Explanation, reason, provider response text, and Python `stdout` | Removed or stored as empty values. |
| Evaluator verdict | Preserve `passed`, `value`, and `type`. Preserve available score, label, confidence, categories, outcome, status, and Guardrail action fields. |

For example, a scrubbed boolean Evaluator can still produce this visible result:

```json JSON theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
  "passed": true,
  "type": "boolean",
  "value": true
}
```

Use the preserved verdict to filter traces and monitor quality without storing the source content used to calculate it.

## Attach evaluators

> [tab] Agent
>
> In the Agent configuration, add Evaluators as input or output evaluators.
>
> * **Input evaluator**: evaluates the input before the model generates a response.
> * **Output evaluator**: evaluates the output after generation.
>
> Set a **Sample Rate** (`1` to `100`%) to control what percentage of requests trigger evaluation. A `100`% sample rate evaluates every request; a `10`% rate evaluates one in ten.
>
> See Configure Evaluators and Guardrails for the full guide.

> [tab] Deployment
>
> Evaluators attach to Deployments with the same input/output stage and a **Sample Rate** of `0` to `100`%.

> [tab] Guardrail Rules
>
> For conditional evaluation, use [Guardrail Rules](/docs/ai-gateway/configuration/guardrail-rules.md) to define CEL-based conditions that determine when evaluators run. Evaluation can be triggered based on metadata, identity, model, or other request attributes.

## Sampling and volume control

Every trace and log is collected in full. Sampling decides which evaluations and actions run against that traffic, not what is stored.

| Where a rate is set | Scale | What it gates |
| - | - | - |
| **Sample Rate** on an Evaluator attached to an Agent or Deployment | Percentage: `1` to `100` on an Agent, `0` to `100` on a Deployment | The share of requests the Evaluator scores. A Guardrail ignores the value and runs on every request. |
| `sample_rate` on a `guardrails[]` entry in a Responses request | Fraction, `0` to `1` | Whether that entry runs. On chat completions the same field is a percentage, `0` to `100`. See [Guardrails](/docs/ai-gateway/configuration/guardrails.md#timeout). |
| **Sample Rate** on a [Trace Automation](/docs/ai-studio/observability/automations.md) | Percentage, `0` to `100` | The share of matching spans whose Action runs. |

Each rate is drawn per request for an Evaluator and per span for an Automation, so a 10% rate keeps roughly one request in ten rather than every tenth request in sequence. A rate applies to traffic that arrives after it is saved; results already recorded stay.

## View evaluator results

Evaluator results attach to individual spans within a trace. Each result includes the Evaluator name, the evaluation score or verdict, and the evaluation stage (`input` or `output`).

In the [Traces](/docs/ai-studio/observability/traces.md) section, evaluator results appear in the trace hierarchy alongside LLM calls, tool invocations, and other spans. Use the **Evals** filter to narrow traces by evaluator name, score, or pass/fail status.

If an evaluator result needs correction, hover the result on a span and select the edit icon. Set the corrected value and optionally add an explanation. Corrections are written back to the span and are queryable via the **Toqan Router MCP**.

## Create an Evaluator

Browse the Marketplace for pre-built evaluators or create a custom one:

* **LLM Evaluator**: uses a model to judge outputs against criteria defined in a prompt.
* **Python Evaluator**: runs custom Python code for deterministic scoring.

See Create Evaluators for the full guide.

> [card] [Trace Automations](/docs/ai-studio/observability/automations.md)
>
> Automatically route traces with low evaluator scores to Annotation Queues.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.