LlamaIndex Agents + Orq: Production observability for LlamaIndex agents

What is LlamaIndex Agents?

LlamaIndex Agents extends the LlamaIndex framework with higher‑level agent abstractions that sit on top of indexes, query engines, tools, and workflows. It helps teams focus on agent behavior over their data instead of manually wiring every retrieval, tool call, and orchestration step.

Why use Orq with LlamaIndex Agents?

Trace LlamaIndex Agent runs end to end, including tool plans, retrieval steps, index lookups, LLM calls, and errors in one place. You get clearer visibility into how each agent decided, retrieved, and answered, without stitching together traces across multiple services.

Test new models, add alternative providers, or assign different model tiers to different agents and tools from Orq.ai, while keeping your LlamaIndex Agent definitions and workflows stable. Your agent code still “thinks” it’s talking to a single LLM interface; Orq handles which underlying model is called.

Use real agent traces like plans, tool call chains, query spans, and LLM interactions to build datasets, compare prompt or model changes, and move from subjective tuning to measurable quality checks. Failed or slow agent runs can be turned into eval inputs before you roll out changes.

Track token usage and spend per agent, tool, team, and workflow so you can see which routes drive cost. Add budgets, rate limits, and approved‑model lists at the Orq platform layer instead of enforcing governance separately inside each LlamaIndex Agents application.

How the integration works

Configure the LLM clients used by your LlamaIndex Agents (for example an OpenAI‑style client) to use Orq.ai’s router as the base URL with your Orq API key and chosen model names. This gives Orq the context it needs to apply routing rules, capture usage, and enforce fallback behavior.

Keep your existing observability setup for LlamaIndex (spans for indexing, retrieval, and workflows) while exporting agent‑level and LLM traces to Orq’s telemetry endpoint. LlamaIndex Agents continues to emit spans for agent steps and tool calls, and Orq records LLM interactions, latency, and cost metrics, so you can see both agent‑level and model‑level traces.

Create routes for key LlamaIndex Agent workflows (for example enterprise QA agents, multi‑tool assistants, or data‑aware copilots) and assign them model tiers, fallback chains, and region/data policies. Agents send LLM requests to the configured Orq routes, and Orq applies routing before invoking OpenAI, Anthropic, Gemini, or other models.

Once connected, use Orq’s dashboards to watch latency, errors, and cost for LlamaIndex Agents, and run evals or experiments on their traces. You can adjust routes, models, or prompt configurations centrally where supported, while LlamaIndex Agents continues to coordinate indexes, query engines, tools, and workflows.

Use cases

Multi‑agent products with real observability

Trace which LlamaIndex Agent, tool, index, and model contributed to a failed answer or hallucination, combining LlamaIndex’s agent/workflow spans with Orq’s router‑level LLM traces.

Cost‑aware internal tools

Route routine “lookup” agents and simple structured Q&A to lower‑cost models while keeping complex reasoning agents or high‑stakes enterprise assistants on your strongest routes configured in Orq.

Eval‑driven agent improvements

Reuse failed conversations, mis‑routed tool calls, and slow agent runs as eval inputs before shipping prompt, retrieval, or model changes, turning production telemetry into a continuous improvement loop.

Safer experimentation across providers

Test a new model or provider on a small share of LlamaIndex Agent traffic, then promote or roll back based on traces and evals, instead of editing each agent or LLM configuration by hand.

With Orq.ai vs without

Capability

LlamaIndex Agents alone

LlamaIndex Agents + Orq.ai

Model access

LLM providers and API keys are configured directly in LlamaIndex LLM clients inside each agent project

Model access is managed through Orq.ai’s router, keeping provider changes and multi‑provider routing outside the agent code

Observability

Teams use LlamaIndex’s own observability integrations to trace agents, indexes, and workflows

Orq centralizes traces for routed LLM traffic, including model requests, responses, errors, and cost metrics alongside LlamaIndex Agents’ spans

Reliability

Retries and fallback behavior live in agent logic, query engine configuration, or custom code

Fallback chains, retries, and routing rules can be configured centrally in Orq and reused across LlamaIndex Agent workflows

Evals

Evaluation uses custom datasets and observability tools to assess agent answer quality and RAG performance

LlamaIndex Agent traces and run outputs can be used as inputs for eval datasets and experiments directly inside Orq.ai as well

Cost & governance

Usage, budgets, and provider policies are spread across LlamaIndex Agent projects and provider accounts

Cost tracking, budgets, access rules, and approved‑model policies can be managed from one control layer in Orq

FAQ

Do I have to change my LlamaIndex Agents code to use Orq.ai?

In many cases, you can repoint the LLM clients your agents use (base URL, key, and model identifier) to Orq’s router and enable trace export to Orq, keeping agent definitions, indexes, query engines, and tools the same. Data connectors and retrieval logic stay in LlamaIndex.

Does Orq.ai replace LlamaIndex Agents or its observability ecosystem?

No. LlamaIndex Agents remains your framework for data‑aware agents, and you can keep using your existing observability tools. Orq.ai sits alongside it as the control plane for models, multi‑provider routing, cost tracking, and additional evaluation.

You still design and run context‑aware agents with LlamaIndex at the core, but you use Orq to see what they cost, which models they should call, and how LLM behavior compares across providers under the same agent workflow.

Can I keep using my existing LLM providers with LlamaIndex Agents if I move to Orq.ai?

Yes. You can bring your existing provider keys into Orq.ai and route LlamaIndex Agent traffic through them, alongside any new models you add later. That way you centralize access, routing, and tracking without losing the LLM and LlamaIndex setup you already rely on.

Bring production controls to LlamaIndex Agents

Move LlamaIndex‑based agents from prototype to production with Orq.ai’s routing, traces, evals, and cost controls already in place.

[Get started in the docs →] [Talk to the team →]