Batches

See HTTP URLs and authentication and the API reference for complete request and response formats. Batches are the best way to process a large volume of requests: they run without the throughput limits of real-time requests, at a fraction of the price. The Batches API runs large offline workloads without a request per line. Upload a JSONL file of requests, create a batch against a supported endpoint, and download the results once the batch reaches a terminal state. The API is OpenAI-compatible at /v3/router/batches. Batch requests, response objects, status values, and result files follow the OpenAI batch format, with Toqan Router additions for the endpoint list, listing filters, and project scoping.

Provider batches run against models that declare batch support for the endpoint and are priced at the provider’s batch rates. Check batch support for a model on the Models page.

Supported endpoints

Endpoint	Runs through	Providers
`/v1/responses`	Provider batch API	OpenAI, Google, Anthropic, AWS Bedrock (Anthropic models)
`/v1/chat/completions`	Provider batch API	OpenAI, Google, Anthropic, AWS Bedrock (Anthropic models), Mistral
`/v1/embeddings`	Provider batch API	OpenAI, Mistral
`/v1/completions`	Provider batch API	OpenAI
`/v1/moderations`	Provider batch API	OpenAI, Mistral
`/v3/router/decisions`	Toqan Router workers	Any model enabled for Decisions

Provider batches run through the provider’s batch service. Decisions batches run on Toqan Router workers, which pace the lines and check budgets per line. The OpenAI SDK type definitions declare /v1/responses, /v1/chat/completions, /v1/embeddings, and /v1/completions. Send /v1/moderations and /v3/router/decisions with the HTTP API.

Run a batch

Create a batch from the console or the API.

AI Studio
API & SDK

Open Batches under AI Gateway. The list shows one row per batch with its ID, status, model, endpoint, request counts, and creation time.
Click Batch.
Choose the JSONL file, pick the endpoint, add metadata when needed, and click Create batch. The completion window is 24h and the field is not editable.
Follow the request counts, files, and status timeline in the list or the batch detail drawer. Filter by Status and Endpoint, or search by batch ID, model, or endpoint.
Download the results once the batch finishes: the drawer footer shows Download results, and Download errors beside it when the batch produced an error file, or on its own when the batch produced no output file. Cancel a running batch from the row actions or the detail drawer.

Write the input file

Write a JSONL file with one request per line. Each line carries a unique custom_id, "method": "POST", a url that matches the batch endpoint, and the request as body.

JSON

{"custom_id": "ticket-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openai/gpt-4o", "messages": [{"role": "user", "content": "Summarize this support ticket in one sentence."}]}}
{"custom_id": "ticket-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openai/gpt-4o", "messages": [{"role": "user", "content": "Draft a reply to this support ticket."}]}}

Upload the input file

Upload the file with the Files API and purpose set to FILE_PURPOSE_BATCH. The returned file ID is the input_file_id for the batch.

curl -X POST https://router.toqan.ai/eu/v2/files \
  -H "Authorization: Bearer $TOQAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"filename\": \"requests.jsonl\",
    \"content\": \"$(base64 < requests.jsonl | tr -d '\n')\",
    \"content_type\": \"application/jsonl\",
    \"purpose\": \"FILE_PURPOSE_BATCH\"
  }"

Create the batch

Send the file ID, the endpoint the lines target, and a completion_window of 24h. Other values are rejected with 400.output_expires_after sets a lifetime for both result files when the batch runs through a provider that supports output expiry: seconds, counted from created_at. An omitted field keeps the provider’s default. Batches the gateway executes itself keep their result files available.

curl -X POST https://router.toqan.ai/eu/v3/router/batches \
  -H "Authorization: Bearer $TOQAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file_01HZXW2K7Y8Q9M0N1P2R3S4T5V",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h"
  }'

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

const batch = await client.batches.create({
  input_file_id: "file_01HZXW2K7Y8Q9M0N1P2R3S4T5V",
  endpoint: "/v1/chat/completions",
  completion_window: "24h",
});

console.log(batch.id);

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["TOQAN_API_KEY"],
    base_url="https://router.toqan.ai/eu/v3/router",
)

batch = client.batches.create(
    input_file_id="file_01HZXW2K7Y8Q9M0N1P2R3S4T5V",
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

print(batch.id)

The creation is validated up front. Every line is checked for a valid custom_id, method, url, and model before the batch is created. An invalid line rejects the file with 400 and names the line.

Poll the batch

Call GET /v3/router/batches/{batch_id} until status is completed, failed, cancelled, or expired. request_counts reports how many lines completed and failed so far.

curl https://router.toqan.ai/eu/v3/router/batches/$BATCH_ID \
  -H "Authorization: Bearer $TOQAN_API_KEY"

import OpenAI from "openai";
import { setTimeout as sleep } from "node:timers/promises";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

let batch = await client.batches.retrieve("batch_01K9X2W7Y8Q9M0N1P2R3S4T5V");

while (!["completed", "failed", "cancelled", "expired"].includes(batch.status)) {
  await sleep(5000);
  batch = await client.batches.retrieve(batch.id);
}

console.log(batch.status, batch.request_counts);

import os
import time
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["TOQAN_API_KEY"],
    base_url="https://router.toqan.ai/eu/v3/router",
)

batch = client.batches.retrieve("batch_01K9X2W7Y8Q9M0N1P2R3S4T5V")

while batch.status not in ("completed", "failed", "cancelled", "expired"):
    time.sleep(5)
    batch = client.batches.retrieve(batch.id)

print(batch.status, batch.request_counts)

Download the results

Call GET /v3/router/batches/{batch_id}/results to download the output file. Send ?file=errors to download the failed and unexecuted lines instead.

cURL

curl https://router.toqan.ai/eu/v3/router/batches/$BATCH_ID/results \
  -H "Authorization: Bearer $TOQAN_API_KEY" \
  -o results.jsonl

curl "https://router.toqan.ai/eu/v3/router/batches/$BATCH_ID/results?file=errors" \
  -H "Authorization: Bearer $TOQAN_API_KEY" \
  -o errors.jsonl

Each download is named after the request: <batch_id>-results.jsonl for the output file, <batch_id>-errors.jsonl for the error file. Without file, the response carries the output file, or the errors file when the batch produced no output, under the -results name. A batch with no error file answers 409 output_unavailable.Both result files expire according to the batch’s output_expires_after; download them before it ends.Downloads are available once the batch reaches a terminal state. Requesting one earlier returns 409 invalid_request.

Input file

Each line follows the OpenAI batch input format.

Field	Type	Description
`custom_id`	string	Required. Unique identifier for the line, echoed in the results.
`method`	string	Required. `POST`.
`url`	string	Required. Must match the batch `endpoint`.
`body`	object	Required. The request body, including `model`.

Limits apply to the whole file and to each line.

Limit	Value
File size	100 MiB
Lines per batch	99,999
Line size	20 MiB
Completion window	`24h`

All lines in a provider batch must resolve to the same provider, and each model must declare batch support for the endpoint. Lines can use different models only when they share a provider and support the endpoint.

Batch statuses

Status	Meaning
`validating`	The batch is being checked before work starts.
`in_progress`	Lines are running.
`finalizing`	All lines finished and the result files are being written.
`completed`	The batch finished. Results are available.
`failed`	The batch stopped with an error. Details are in `errors`.
`expired`	The 24-hour completion window passed before all lines ran.
`cancelling`	A cancel request is being applied.
`cancelled`	The batch stopped. Lines that had not run appear in the errors file.

List and filter batches

GET /v3/router/batches returns batches in the caller’s projects, newest first.

Parameter	Type	Description
`limit`	integer	Batches per page, 1 to 100. Defaults to 20.
`after`	string	Return batches created before this batch ID.
`before`	string	Return batches created after this batch ID. Cannot be combined with `after`.
`status`	string	Comma-separated statuses, for example `completed,failed`.
`endpoint`	string	Comma-separated endpoints.
`search`	string	Case-insensitive match on batch ID, model, or endpoint.

curl "https://router.toqan.ai/eu/v3/router/batches?limit=20" \
  -H "Authorization: Bearer $TOQAN_API_KEY"

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

const page = await client.batches.list({ limit: 20 });
console.log(page.data.length, page.has_more);

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["TOQAN_API_KEY"],
    base_url="https://router.toqan.ai/eu/v3/router",
)

page = client.batches.list(limit=20)
print(len(page.data), page.has_more)

The OpenAI SDK list method sends limit and after. Send before, status, endpoint, and search over the HTTP API:

cURL

curl "https://router.toqan.ai/eu/v3/router/batches?limit=20&status=completed&endpoint=/v1/chat/completions&search=gpt-4o" \
  -H "Authorization: Bearer $TOQAN_API_KEY"

The response uses the OpenAI cursor list envelope:

JSON

{
  "object": "list",
  "data": [
    {
      "id": "batch_01K9X2W7Y8Q9M0N1P2R3S4T5V",
      "object": "batch",
      "endpoint": "/v1/chat/completions",
      "input_file_id": "file_01HZXW2K7Y8Q9M0N1P2R3S4T5V",
      "completion_window": "24h",
      "status": "completed",
      "output_file_id": "file_01K9X2W7Y8Q9M0N1P2R3S4T5W",
      "created_at": 1761000000,
      "request_counts": { "total": 2, "completed": 2, "failed": 0 },
      "model": "openai/gpt-4o"
    }
  ],
  "first_id": "batch_01K9X2W7Y8Q9M0N1P2R3S4T5V",
  "last_id": "batch_01K9X2W7Y8Q9M0N1P2R3S4T5V",
  "has_more": false
}

Results

Each result line follows the OpenAI batch output format. response.body is the endpoint response for that line. response.request_id is the provider’s response identifier for a provider batch, and the trace ID of the line for a Decisions batch. A provider batch records one trace for the whole batch when it reaches a terminal state, so look it up by the batch ID: the trace carries orq.batch.id. A Decisions batch traces each line separately, and that trace ID is the line’s request_id.

JSON

{"id": "batch_req_01K9X2W7Y8Q9M0N1P2R3S4T5V", "custom_id": "ticket-1", "response": {"status_code": 200, "request_id": "4f2294c4bd55372a354be496871aa917", "body": {"id": "chatcmpl-01K9X2W7Y8Q9M0N1P2R3S4T5W", "model": "openai/gpt-4o", "choices": [{"index": 0, "message": {"role": "assistant", "content": "The customer was charged twice for the same order."}, "finish_reason": "stop"}]}}, "error": null}

Lines that failed carry an error object instead of a response:

JSON

{"id": "batch_req_01K9X2W7Y8Q9M0N1P2R3S4T5W", "custom_id": "ticket-2", "response": null, "error": {"code": "batch_expired", "message": "The line did not run before the completion window expired."}}

Code	Meaning
`batch_cancelled`	The batch was cancelled before the line ran.
`batch_expired`	The line did not run within the 24-hour completion window.

Provider batches can also return provider-specific error codes for lines the provider rejected. Both files download from GET /v3/router/batches/{batch_id}/results, as shown in the download step under Run a batch: the output file by default, the error file with ?file=errors.

Behaviour

Validation up front: every line is checked before the batch is created, including the model and its access checks. An invalid line rejects the whole file with 400 and names the line.
Tracing and billing: a provider batch records one trace with the batch’s aggregate usage when it reaches a terminal state, priced at the provider’s batch rates. A Decisions batch runs on Toqan Router workers, and each line is traced and billed like a single request.
Budgets: Decisions batches check budgets per line. A line that hits a cost or token budget fails with 429 in its result; a requests-per-minute budget delays the line for up to two minutes before failing it.
Project scoping: a batch belongs to the project of its input file. List, retrieve, download, and cancel only reach batches in the caller’s projects.
Permissions: the API key needs the batches permission. A Decisions batch also needs the classify permission.
Rate limits: creating a batch counts as one request against plan rate limits. See Rate limits and quotas.
Cancel: POST /v3/router/batches/{batch_id}/cancel stops the lines that have not run. Calling it on a terminal batch returns the stored batch unchanged.

Decisions

Batch classification requests against a Decisions model.

Background Responses

Run a single long-running response without batching.

Toqan Router

Overview

APIs

MCP Portal

Routing

Security & Privacy

Model Capabilities

Server Tools

Common Scenarios

Provider setup

Supported endpoints

Run a batch

Input file

Batch statuses

List and filter batches

Results

Behaviour

Decisions

Background Responses

​Supported endpoints

​Run a batch

​Input file

​Batch statuses

​List and filter batches

​Results

​Behaviour

​Related

Decisions

Background Responses

Supported endpoints

Run a batch

Input file

Batch statuses

List and filter batches

Results

Behaviour

Related