LlamaIndex framework integration

AI Gateway

Route your LLM calls through the AI Gateway with a single base URL change. Zero vendor lock-in: always run on the best model at the lowest cost for your use case.

Observability

Instrument your code with OpenTelemetry to capture traces, logs, and metrics for every LLM call, agent step, and tool use.

AI Gateway

Overview

LlamaIndex is a powerful framework for building RAG (Retrieval-Augmented Generation) applications with LLMs. By connecting LlamaIndex to Toqan Router’s AI Gateway, you transform experimental RAG applications into production-ready systems with enterprise-grade capabilities.

Key Benefits

Toqan Router’s AI Gateway enhances your LlamaIndex applications with:

Complete Observability

Track document indexing, retrieval performance, and query processing with detailed traces

Built-in Reliability

Automatic fallbacks, retries, and load balancing for production resilience

Cost Optimization

Real-time cost tracking and spend management across all your AI operations

Multi-Provider Access

Access 500+ LLMs and 30+ providers through a single, unified integration

Prerequisites

Before integrating LlamaIndex with Toqan Router, ensure you have:

A Toqan Router account and API Key
Python 3.8 or higher
LlamaIndex installed in your project

To setup your API key, see API keys & Endpoints.

Installation

Install LlamaIndex and required dependencies:

pip install llama-index llama-index-llms-openai-like

Configuration

Configure LlamaIndex to use Toqan Router’s AI Gateway with the OpenAILike class:

Python

from llama_index.llms.openai_like import OpenAILike
import os

# Configure OpenAI-compatible LLM with Toqan Router AI Gateway
llm = OpenAILike(
    model="openai/gpt-5.6-sol",
    api_key=os.getenv("TOQAN_API_KEY"),
    api_base="https://router.toqan.ai/eu/v3/router",
    is_chat_model=True,
)

api_base: https://router.toqan.ai/eu/v3/router

Basic RAG Example

Here’s a complete example of building a RAG application with LlamaIndex through Toqan Router:

Python

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai_like import OpenAILike
from llama_index.embeddings.openai import OpenAIEmbedding
import os

# Configure LLM with Toqan Router AI Gateway
llm = OpenAILike(
    model="openai/gpt-5.6-sol",
    api_key=os.getenv("TOQAN_API_KEY"),
    api_base="https://router.toqan.ai/eu/v3/router",
    is_chat_model=True,
)

# Configure embeddings through Toqan Router (required - LlamaIndex defaults to OpenAI)
embed_model = OpenAIEmbedding(
    model="openai/text-embedding-3-small",
    api_key=os.getenv("TOQAN_API_KEY"),
    api_base="https://router.toqan.ai/eu/v3/router",
)

# Set as global defaults
Settings.llm = llm
Settings.embed_model = embed_model

# Load documents and create index
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)

# Query the index
query_engine = index.as_query_engine()
response = query_engine.query("What is the main topic of these documents?")
print(response)

Model Selection

With Toqan Router, you can use any supported model from 30+ providers:

Python

from llama_index.llms.openai_like import OpenAILike
import os

# Use Claude
claude_llm = OpenAILike(
    model="anthropic/claude-sonnet-5",
    api_key=os.getenv("TOQAN_API_KEY"),
    api_base="https://router.toqan.ai/eu/v3/router",
    is_chat_model=True,
)

# Use Gemini
gemini_llm = OpenAILike(
    model="google-ai/gemini-2.5-flash",
    api_key=os.getenv("TOQAN_API_KEY"),
    api_base="https://router.toqan.ai/eu/v3/router",
    is_chat_model=True,
)

# Use any other model
groq_llm = OpenAILike(
    model="groq/qwen/qwen3.8-27b",
    api_key=os.getenv("TOQAN_API_KEY"),
    api_base="https://router.toqan.ai/eu/v3/router",
    is_chat_model=True,
)

Streaming Responses

LlamaIndex supports streaming with Toqan Router:

Python

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai_like import OpenAILike
from llama_index.embeddings.openai import OpenAIEmbedding
import os

# Configure LLM
llm = OpenAILike(
    model="openai/gpt-5.6-sol",
    api_key=os.getenv("TOQAN_API_KEY"),
    api_base="https://router.toqan.ai/eu/v3/router",
    is_chat_model=True,
)

# Configure embeddings
embed_model = OpenAIEmbedding(
    model="openai/text-embedding-3-small",
    api_key=os.getenv("TOQAN_API_KEY"),
    api_base="https://router.toqan.ai/eu/v3/router",
)

Settings.llm = llm
Settings.embed_model = embed_model

# Create index and query engine
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(streaming=True)

# Stream response
streaming_response = query_engine.query("Explain the main concepts")
for text in streaming_response.response_gen:
    print(text, end="", flush=True)
print()

Observability

Getting Started

Integrate LlamaIndex with Toqan Router’s observability to gain comprehensive insights into document indexing, retrieval performance, query processing, and LLM interactions using OpenTelemetry.

Prerequisites

Before you begin, ensure you have:

A Toqan Router account and API Key
LlamaIndex installed in your project
Python 3.8+
OpenAI API key (or other LLM provider credentials)

Install Dependencies

# Core LlamaIndex and OpenInference packages
pip install llama-index openinference-instrumentation-llama-index

# OpenTelemetry packages
pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

# LLM providers
pip install openai anthropic

# Optional: For advanced vector stores and embeddings
pip install llama-index-vector-stores-chroma llama-index-embeddings-openai

Configure Toqan Router

Set up your environment variables to connect to Toqan Router’s OpenTelemetry collector: Unix/Linux/macOS:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://router.toqan.ai/eu/v2/otel"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <TOQAN_API_KEY>"
export OTEL_RESOURCE_ATTRIBUTES="service.name=llamaindex-app,service.version=1.0.0"
export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"

Windows (PowerShell):

$env:OTEL_EXPORTER_OTLP_ENDPOINT = "https://router.toqan.ai/eu/v2/otel"
$env:OTEL_EXPORTER_OTLP_HEADERS = "Authorization=Bearer <TOQAN_API_KEY>"
$env:OTEL_RESOURCE_ATTRIBUTES = "service.name=llamaindex-app,service.version=1.0.0"
$env:OPENAI_API_KEY = "<YOUR_OPENAI_API_KEY>"

Using .env file:

OTEL_EXPORTER_OTLP_ENDPOINT=https://router.toqan.ai/eu/v2/otel
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <TOQAN_API_KEY>
OTEL_RESOURCE_ATTRIBUTES=service.name=llamaindex-app,service.version=1.0.0
OPENAI_API_KEY=<YOUR_OPENAI_API_KEY>

Integration Example

We’ll be using OpenInference as TracerProvider with LlamaIndex

Python

from openinference.instrumentation.llama_index import LlamaIndexInstrumentor
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk import trace as trace_sdk
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

# Initialize OpenTelemetry
tracer_provider = trace_sdk.TracerProvider()
tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
    endpoint="https://router.toqan.ai/eu/v2/otel/v1/traces",
    headers={"Authorization": "Bearer <TOQAN_API_KEY>"}
)))
trace.set_tracer_provider(tracer_provider)

# Instrument LlamaIndex
LlamaIndexInstrumentor().instrument(tracer_provider=tracer_provider)

# Your LlamaIndex code is automatically traced
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(documents)

query_engine = index.as_query_engine()
response = query_engine.query("What is the main topic of these documents?")

View Traces

View your traces in the AI Studio in the Traces tab.

Visit your AI Studio to view real-time analytics and traces.

Overview

Instrument your app

Explore

Monitor

Export

Review

Data privacy & security

Framework integrations

AI Gateway

Observability

AI Gateway

Overview

Key Benefits

Complete Observability

Built-in Reliability

Cost Optimization

Multi-Provider Access

Prerequisites

Installation

Configuration

Basic RAG Example

Model Selection

Streaming Responses

Observability

Getting Started

Prerequisites

Install Dependencies

Configure Toqan Router

Integration Example

View Traces

AI Gateway

Observability

​AI Gateway

​Overview

​Key Benefits

Complete Observability

Built-in Reliability

Cost Optimization

Multi-Provider Access

​Prerequisites

​Installation

​Configuration

​Basic RAG Example

​Model Selection

​Streaming Responses

​Observability

​Getting Started

​Prerequisites

​Install Dependencies

​Configure Toqan Router

​Integration Example

​View Traces

AI Gateway

Overview

Key Benefits

Prerequisites

Installation

Configuration

Basic RAG Example

Model Selection

Streaming Responses

Observability

Getting Started

Prerequisites

Install Dependencies

Configure Toqan Router

Integration Example

View Traces