> ## Documentation Index
> Fetch the complete documentation index at: /docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Fallbacks and retries in the AI Gateway

> Retry failed LLM requests with exponential backoff and configure fallback models in Toqan Router to handle rate limits, server errors, and network failures.

**Use Cases**

* Surviving transient provider errors without surfacing failures to end users.
* Automatic failover to a backup provider when the primary is degraded or rate-limited.
* Absorbing short rate-limit bursts without manual intervention or custom retry logic.
* Meeting availability SLAs on production features without adding retry code to every service.

***

> [card] [Retries](#retries)
>
> Retry failed requests automatically with exponential backoff. Configure which HTTP error codes trigger retries and how many attempts to make.

> [card] [Fallbacks](#fallbacks)
>
> Route to a different model when the primary fails. Define a fallback chain across providers for high availability.

## Retries

Automatically retry failed requests with exponential backoff.

### Quick Start

```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://router.toqan.ai/eu/v3/router/responses \
  -H "Authorization: Bearer $TOQAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5.4-mini",
    "input": "Analyze customer feedback",
    "retry": {"count": 3, "on_codes": [429, 500, 502, 503, 504]}
  }'
```

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

const response = await client.responses.create({
  model: "openai/gpt-5.4-mini",
  input: "Analyze customer feedback",
  retry: {
    count: 3,
    on_codes: [429, 500, 502, 503, 504],
  },
});

console.log(response.output_text);
```

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("TOQAN_API_KEY"),
    base_url="https://router.toqan.ai/eu/v3/router",
)

response = client.responses.create(
    model="openai/gpt-5.4-mini",
    input="Analyze customer feedback",
    extra_body={
        "retry": {"count": 3, "on_codes": [429, 500, 502, 503, 504]}
    },
)

print(response.output_text)
```

```typescript TypeScript (Chat Completions) theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

const response = await client.chat.completions.create({
  model: "openai/gpt-5.4-mini",
  messages: [{ role: "user", content: "Analyze customer feedback" }],
  retry: {
    count: 3,
    on_codes: [429, 500, 502, 503, 504],
  },
});
```

### Configuration

| Parameter | Type | Required | Description |
| - | - | - | - |
| `count` | number | Yes | Retries per model after the initial attempt (0-5). `0` disables retries. |
| `on_codes` | number\[] | No | HTTP status codes that trigger retries (default: \[408, 429, 500, 502, 503, 504, 529]) |

When `retry` is omitted, the gateway retries twice on the default codes. Set `"retry": {"count": 0}` to make exactly one call per model.

### Error Codes

| Code | Meaning | Retry? | Common Cause |
| - | - | - | - |
| `408` | Request timeout |  Yes | Provider timeout |
| `429` | Rate limit exceeded |  Yes | Too many requests |
| `500` | Internal server error |  Yes | Provider issue |
| `501` | Not implemented |  No | Definitive; retrying will not succeed |
| `502` | Bad gateway |  Yes | Network/Gateway issue |
| `503` | Service unavailable |  Yes | Provider maintenance |
| `504` | Gateway timeout |  Yes | Provider overload |
| `529` | Provider overloaded |  Yes | Provider overload |
| `400` | Bad request |  No | Invalid parameters |
| `401` | Unauthorized |  No | Invalid API key |
| `403` | Forbidden |  No | Access denied |

### Retry Strategies

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
// Conservative
retry: {
  count: 2,
  on_codes: [429, 503]  // Only rate limits and service unavailable
}

// Balanced (recommended)
retry: {
  count: 3,
  on_codes: [429, 500, 502, 503, 504]  // All transient errors
}

// Aggressive
retry: {
  count: 5,
  on_codes: [429, 500, 502, 503, 504]  // Max retries
}
```

### Backoff Algorithm

#### Exponential backoff with jitter

* Attempt 1: 1s (±25%).
* Attempt 2: 2s (±25%).
* Attempt 3: 4s (±25%).
* Attempt 4: 8s (±25%).
* Attempt 5: 16s (±25%).
  **Maximum total delay**: \~31 seconds for 5 retries

### Code examples

```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://router.toqan.ai/eu/v3/router/responses \
  -H "Authorization: Bearer $TOQAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5.4-mini",
    "input": "Analyze customer feedback and provide sentiment analysis",
    "retry": {
      "count": 3,
      "on_codes": [429, 500, 502, 503, 504]
    }
  }'
```

```bash cURL (Chat Completions) theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://router.toqan.ai/eu/v3/router/chat/completions \
  -H "Authorization: Bearer $TOQAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5.4-mini",
    "messages": [
      {
        "role": "user",
        "content": "Analyze customer feedback and provide sentiment analysis"
      }
    ],
    "retry": {
      "count": 3,
      "on_codes": [429, 500, 502, 503, 504]
    }
  }'
```

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

const response = await client.responses.create({
  model: "openai/gpt-5.4-mini",
  input: "Analyze customer feedback and provide sentiment analysis",
  retry: {
    count: 3,
    on_codes: [429, 500, 502, 503, 504],
  },
});

console.log(response.output_text);
```

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("TOQAN_API_KEY"),
    base_url="https://router.toqan.ai/eu/v3/router",
)

response = client.responses.create(
    model="openai/gpt-5.4-mini",
    input="Analyze customer feedback and provide sentiment analysis",
    extra_body={
        "retry": {
            "count": 3,
            "on_codes": [429, 500, 502, 503, 504],
        }
    },
)

print(response.output_text)
```

```typescript TypeScript (Chat Completions) theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

const response = await client.chat.completions.create({
  model: "openai/gpt-5.4-mini",
  messages: [
    {
      role: "user",
      content: "Analyze customer feedback and provide sentiment analysis",
    },
  ],
  retry: {
    count: 3,
    on_codes: [429, 500, 502, 503, 504],
  },
});
```

```python Python (Chat Completions) theme={"theme":{"light":"github-light","dark":"github-dark"}}
from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("TOQAN_API_KEY"),
    base_url="https://router.toqan.ai/eu/v3/router",
)

response = client.chat.completions.create(
    model="openai/gpt-5.4-mini",
    messages=[
        {
            "role": "user",
            "content": "Analyze customer feedback and provide sentiment analysis",
        }
    ],
    extra_body={
        "retry": {
            "count": 3,
            "on_codes": [429, 500, 502, 503, 504],
        }
    },
)
```

### Best Practices

#### Production recommendations

Follow this advice for a solid production setup:

* Use `count: 2-3` for balance of reliability and speed.
* Always include `429` (rate limits) in `on_codes`.
* Monitor retry rates to detect systemic issues.
* Implement a circuit breaker for persistent failures.

#### Error handling

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

try {
  const response = await client.responses.create({
    model: "openai/gpt-5.4-mini",
    input: "Hello",
  });
} catch (error) {
  if (error instanceof OpenAI.APIError) {
    if (error.status === 400) {
      console.error('Bad request:', error.message);
    } else if (error.status >= 500) {
      console.error('Server error:', error.message);
    }
  }
}
```

```typescript TypeScript (Chat Completions) theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

try {
  const response = await client.chat.completions.create({
    model: "openai/gpt-5.4-mini",
    messages: [{ role: "user", content: "Hello" }],
  });
} catch (error) {
  if (error instanceof OpenAI.APIError) {
    if (error.status === 400) {
      // Don't retry client errors - fix the request
      console.error('Bad request:', error.message);
    } else if (error.status >= 500) {
      // Server errors might need manual intervention
      console.error('Server error:', error.message);
    }
  }
}
```

### Troubleshooting

**High retry rates**

* Check whether rate limits are being hit frequently.
* Verify API keys have sufficient quotas.
* Monitor provider status pages for outages.

**Slow response times**

* Reduce retry count for latency-sensitive apps.
* Use shorter timeout values with retries.
* Consider fallbacks for faster alternatives.

**Still getting errors**

* Check if error codes are in `on_codes` list.
* Verify retry count isn't exhausted.
* Review provider-specific error documentation.

### Monitoring

Track these retry metrics:

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const retryMetrics = {
  totalRequests: 0,
  retriedRequests: 0,
  retriesByAttempt: { 1: 0, 2: 0, 3: 0 }, // Retry attempt distribution
  retriesByCode: { 429: 0, 500: 0 }, // By error code
  avgRetryLatency: 0, // Added latency from retries
  finalFailures: 0, // Requests that failed after all retries
};
```

### Limitations

* **Increased latency**: Retries add delay (up to 31s for 5 retries).
* **Cost implications**: Failed requests may still incur charges.
* **Rate limit consumption**: Retries run inside the gateway on one incoming request, so a retry counts once against platform rate limits and budgets, while every attempt counts against the provider's own rate limits.
* **Limited retries**: Maximum 5 retries per model after the initial attempt (up to 6 calls) to prevent excessive delays.
* **Non-retryable errors**: a status code is retried only when it is listed in `on_codes`. Most 4xx client errors, such as 400 and 403, are not retried; 429 is retried when it is listed in `on_codes`.

### Advanced Usage

**Environment-specific configs:**

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const retryConfig = {
  development: { count: 1, on_codes: [429] }, // Fast feedback
  staging: { count: 2, on_codes: [429, 503] }, // Light retries
  production: { count: 3, on_codes: [429, 500, 502, 503, 504] }, // Full protection
};
```

**With other features:**

```json JSON theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
  "retry": { "count": 3, "on_codes": [429, 503] },
  "timeout": { "call_timeout": 10000 },
  "fallbacks": [{ "model": "backup-model" }],
  "cache": { "type": "exact_match", "ttl": 300 }
}
```

**Custom retry logic (client-side):**

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const customRetry = async (requestFn, maxAttempts = 3) => {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await requestFn();
    } catch (error) {
      if (attempt === maxAttempts || error.status < 500) {
        throw error; // Final attempt or non-retryable error
      }
      await new Promise(
        (resolve) => setTimeout(resolve, Math.pow(2, attempt) * 1000), // Exponential backoff
      );
    }
  }
};
```

## Fallbacks

Automatically switch to a different model when the primary fails.

### Quick Start

```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://router.toqan.ai/eu/v3/router/chat/completions \
  -H "Authorization: Bearer $TOQAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5.4-mini",
    "messages": [{ "role": "user", "content": "Generate a product description" }],
    "fallbacks": [{ "model": "openai/gpt-5.6-sol" }, { "model": "azure/gpt-5.6-sol" }]
  }'
```

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

const response = await client.responses.create({
  model: "openai/gpt-5.4-mini",
  input: "Generate a product description",
  fallbacks: [{ model: "openai/gpt-5.6-sol" }, { model: "azure/gpt-5.6-sol" }],
});

console.log(response.output_text);
```

```typescript TypeScript (Chat Completions) theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

const response = await client.chat.completions.create({
  model: "openai/gpt-5.4-mini",
  messages: [{ role: "user", content: "Generate a product description" }],
  fallbacks: [{ model: "openai/gpt-5.6-sol" }, { model: "azure/gpt-5.6-sol" }],
});
```

### Configuration

| Parameter | Type | Required | Description |
| - | - | - | - |
| `fallbacks` | Array | Yes | List of fallback models in order of preference |
| `model` | string | Yes | Model identifier for each fallback |
| `fallback_on_codes` | number\[] | No | Status codes that trigger fallback. Default: `[408, 429, 500, 502, 503, 504, 529]`. Empty uses the default. |

Set `fallback_on_codes` at the top level of the request. A custom list **replaces the default codes**. For example, include `400` alongside the defaults to fall back on bad-request errors:

```json JSON theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
  "model": "google/eu.claude-opus-5-5",
  "input": "Hello",
  "fallbacks": [{ "model": "google/gemini-2.5-pro" }],
  "fallback_on_codes": [400, 408, 429, 500, 502, 503, 504, 529]
}
```

Setting `"fallback_on_codes": [400]` alone limits fallback to `400`; rate limits and server errors no longer trigger it. Omit the field or use an empty array to restore the defaults.

### Trigger Conditions

By default, fallbacks activate on the errors below. Override the set with `fallback_on_codes` on a request or in a Fallback routing rule's `models_config`. Supply up to 20 unique integers from 400 through 599. A matching routing rule replaces request-level fallback models and codes.

For example, `"fallback_on_codes": [429, 503]` limits fallback to rate limits and service unavailability. Cancelled requests never fall back. Models missing a required capability are skipped regardless of the configured codes.

| Error Code | Description | Triggers Fallback by Default |
| - | - | - |
| `408` | Request timeout | Yes |
| `429` | Rate limit exceeded | Yes |
| `500` | Internal server error | Yes |
| `501` | Not implemented | No |
| `502` | Bad gateway | Yes |
| `503` | Service unavailable | Yes |
| `504` | Gateway timeout | Yes |
| `529` | Provider overloaded | Yes |
| `404` | Not found | No |
| `400` | Bad request | No |
| `401` | Unauthorized | No |
| `403` | Forbidden | No |

### Best Practices

Use a maximum of 3 fallback models. Order them by preference or cost, and choose models with similar capabilities.

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
// Cost-optimized: cheap then expensive
fallbacks: [{ model: "openai/gpt-5.4-mini" }, { model: "openai/gpt-5.6-sol" }];

// Reliability-optimized: different providers
fallbacks: [
  { model: "openai/gpt-5.6-sol" },
  { model: "anthropic/claude-sonnet-5" },
  { model: "azure/gpt-5.6-sol" },
];
```

### Code examples

```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://router.toqan.ai/eu/v3/router/responses \
  -H "Authorization: Bearer $TOQAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5.4-mini",
    "input": "Generate a product description",
    "fallbacks": [
      { "model": "openai/gpt-5.6-sol" },
      { "model": "azure/gpt-5.6-sol" }
    ]
  }'
```

```bash cURL (Chat Completions) theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://router.toqan.ai/eu/v3/router/chat/completions \
  -H "Authorization: Bearer $TOQAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5.4-mini",
    "messages": [{ "role": "user", "content": "Generate a product description" }],
    "fallbacks": [
      { "model": "openai/gpt-5.6-sol" },
      { "model": "azure/gpt-5.6-sol" }
    ]
  }'
```

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

const response = await client.responses.create({
  model: "openai/gpt-5.4-mini",
  input: "Generate a product description",
  fallbacks: [{ model: "openai/gpt-5.6-sol" }, { model: "azure/gpt-5.6-sol" }],
});

console.log(response.output_text);
```

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("TOQAN_API_KEY"),
    base_url="https://router.toqan.ai/eu/v3/router",
)

response = client.responses.create(
    model="openai/gpt-5.4-mini",
    input="Generate a product description",
    extra_body={
        "fallbacks": [
            {"model": "openai/gpt-5.6-sol"},
            {"model": "azure/gpt-5.6-sol"}
        ]
    }
)

print(response.output_text)
```

```typescript TypeScript (Chat Completions) theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.TOQAN_API_KEY,
  baseURL: "https://router.toqan.ai/eu/v3/router",
});

const response = await client.chat.completions.create({
  model: "openai/gpt-5.4-mini",
  messages: [{ role: "user", content: "Generate a product description" }],
  fallbacks: [{ model: "openai/gpt-5.6-sol" }, { model: "azure/gpt-5.6-sol" }],
});
```

```python Python (Chat Completions) theme={"theme":{"light":"github-light","dark":"github-dark"}}
from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("TOQAN_API_KEY"),
    base_url="https://router.toqan.ai/eu/v3/router",
)

response = client.chat.completions.create(
    model="openai/gpt-5.4-mini",
    messages=[{"role": "user", "content": "Generate a product description"}],
    extra_body={
        "fallbacks": [
            {"model": "openai/gpt-5.6-sol"},
            {"model": "azure/gpt-5.6-sol"}
        ]
    }
)
```

### Limitations

* **Response consistency**: Different models may return varying output styles.
* **Parameter support**: Not all providers support identical parameters.
* **Cost implications**: Failed requests may still incur charges from the primary provider.
* **Latency impact**: Sequential attempts add processing time.
* **Provider dependencies**: Requires API keys for all fallback providers.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.