Use Orq.ai as the model control layer for LlamaIndex. Route LLM calls through one OpenAI‑compatible endpoint, capture traces, monitor cost, and manage fallback behavior without rebuilding your LlamaIndex indexes, query engines, agents, or workflows.
What is LlamaIndex?
LlamaIndex is an open‑source data and agent framework for building context‑aware AI over private and enterprise data, with components for data connectors, indexes, query engines, agents, and workflows. It helps teams focus on RAG design and agent behavior instead of manually wiring every retrieval, indexing, and observability piece.
Why use Orq with LlamaIndex
Trace RAG and agent behavior end to end
Trace LlamaIndex runs end to end, including data ingestion, indexing, retrieval, query processing, tool calls, and LLM responses in one place. You get clearer visibility into how each query and agent behaved, without stitching together traces across multiple services.
Model flexibility without rewiring query engines
Test new models, add alternative providers, or assign different model tiers to different LlamaIndex query engines and agents from Orq.ai, while keeping your LlamaIndex code stable. Your OpenAILike or other LLM interfaces still “think” they are talking to a single client; Orq handles which underlying model is called.
Evaluate real production runs
Use real traces like indexes, query spans, workflow steps, and LLM interactions to build datasets, compare prompt or model changes, and move from subjective tuning to measurable quality checks. Failed or slow queries can be turned into eval inputs before you roll out changes.glassbrain+2
Control spend and access centrally
Track token usage and spend per index, query engine, agent, team, and workflow so you can see which routes drive cost. Add budgets, rate limits, and approved‑model lists at the Orq platform layer instead of enforcing governance separately inside each LlamaIndex project.
How the integration works
Step 1
Point LlamaIndex LLMs at Orq.ai’s router
Configure LLM clients such as OpenAILike to use Orq.ai’s AI Router as their api_base, with your Orq API key and model name (for example gpt-4o). This gives Orq the context it needs to apply routing rules, capture usage, and enforce fallback behavior.
Step 2
Enable tracing from LlamaIndex to Orq
Instrument LlamaIndex with OpenInference and OpenTelemetry, using LlamaIndexInstrumentor or the OpenInference callback handlers, and set OTEL environment variables to export spans to Orq’s OTLP endpoint. LlamaIndex continues to emit spans for ingestion, indexing, retrieval, and LLM calls, and Orq records LLM interactions, latency, and cost metrics.
Step 3
Define routes, fallbacks, and policies in Orq
Create routes for key LlamaIndex workflows (for example production RAG query engines, multi‑tool agents, or workflows) and assign them model tiers, fallback chains, and region/data policies. LlamaIndex sends LLM requests to the configured Orq routes, and Orq applies routing before invoking OpenAI, Anthropic, Gemini, or other models.
Step 4
Monitor, evaluate, and tune
Once connected, use Orq’s dashboards to watch latency, errors, and cost for LlamaIndex‑based RAG and agents, and run evals or experiments on their traces. You can adjust routes, models, or prompt configurations centrally where supported, while LlamaIndex continues to manage data connectors, indexes, query engines, and agents.
Use Cases
Multi‑agent products with real observability
Trace which index, retriever, query engine, and model contributed to a failed answer or hallucination, combining LlamaIndex’s workflow spans with Orq’s router‑level LLM traces.
Cost‑aware internal tools
Route routine “lookup” queries and simple Q&A to lower‑cost models while keeping complex multi‑hop reasoning or high‑stakes enterprise QA on your strongest routes configured in Orq.
Eval‑driven agent improvements
Reuse failed answers, slow queries, and workflow traces as eval inputs before shipping prompt, retrieval, or model changes, turning production telemetry into a continuous improvement loop.
Safer experimentation across providers
Test a new model or provider on a small share of LlamaIndex traffic, then promote or roll back based on traces and evals, instead of editing each query engine or LLM configuration by hand.
FAQ
Do I have to change my LlamaIndex code to use Orq.ai?
In many cases, you can repoint OpenAILike (or similar) LLM clients to Orq’s router (api_base=”https://api.orq.ai/v2/router”) and enable OTEL export to Orq, keeping your indexes, query engines, and agents the same. Data connectors, vector stores, and retrieval logic stay in LlamaIndex.
Does Orq.ai replace LlamaIndex’s own runtime or framework?
No. LlamaIndex remains your framework for RAG and agents, and you can keep using Phoenix, Glassbrain, or OpenTelemetry integrations. Orq.ai sits alongside them as the control plane for models, multi‑provider routing, cost tracking, and additional evaluation.developers.llamaindex+2 You still design and run context‑aware agents and RAG systems with LlamaIndex, but you use Orq to see what they cost, which models they should call, and how LLM behavior compares across providers under the same data‑aware workflow.
Can I keep using my existing LLM providers with LlamaIndex if I move to Orq.ai?
Yes. You can bring your existing provider keys (OpenAI, Anthropic, local models, etc.) into Orq.ai and route LlamaIndex traffic through them, alongside any new models you add later. That way you centralize access, routing, and tracking without losing your current LlamaIndex integrations. Bring production controls to LlamaIndex apps
