Use Orq.ai as the model control layer for Haystack. Route LLM calls through one OpenAI‑compatible endpoint, capture traces, monitor cost, and manage fallback behavior without rebuilding your Haystack pipelines, agent graphs, or component wiring.
What is Haystack?
Haystack is an open‑source AI orchestration framework from deepset for building production‑ready LLM applications as explicit pipelines and agents, with modular components for retrievers, generators, rankers, tools, and document stores. It helps teams focus on pipeline and agent design instead of wiring every integration and observability piece from scratch.
Why use Orq with Haystack
Trace agent and pipeline behavior end to end
Trace Haystack runs end to end, including prompts, retrieval steps, tool invocations, generator calls, and errors in one place. You get clearer visibility into agent and pipeline behavior without stitching together logs from each component.
Model flexibility without rewiring pipelines
Test new models, add alternative providers, or assign different model tiers to different Haystack components from Orq.ai, while keeping your pipelines and agent definitions stable. Your Haystack OpenAIChatGenerator or other LLM components still “think” they are calling a standard LM; Orq handles which underlying model is used.
Evaluate real production runs
Use real Haystack traces—pipeline runs, agent conversations, and retrieval metrics—to build datasets, compare prompt or model changes, and move from subjective tuning to measurable quality checks. Failed or slow runs can be turned into eval inputs before you roll out changes.
Control spend and access centrally
Track token usage and spend per pipeline, agent, team, and workflow so you can see which routes drive cost. Add budgets, rate limits, and approved‑model lists at the Orq platform layer instead of enforcing governance separately inside each Haystack project.
How the integration works
Step 1
Point Haystack’s LLM components at Orq.ai’s router
Configure LLM components like OpenAIChatGenerator, AzureChatGenerator, or compatible generators to use Orq.ai’s base URL and API key instead of calling each provider directly. This gives Orq the context it needs to apply routing rules, capture usage, and enforce fallback behavior.
Step 2
Enable tracing from Haystack to Orq
Keep Haystack’s own observability integrations (for example OpenLIT or OpenTelemetry) while routing LLM calls through Orq’s AI Gateway / Router. Haystack continues to emit spans and metrics for pipeline and agent execution, and Orq records LLM interactions, latency, and cost metrics, so you can see both component‑level and model‑level traces.
Step 3
Define routes, fallbacks, and policies in Orq
Create routes for key Haystack workflows (for example RAG pipelines, QA agents, or tool‑using agents) and assign them model tiers, fallback chains, and region/data policies. Haystack sends LLM requests to the configured Orq routes, and Orq applies routing before invoking OpenAI, Gemini, Claude, or other models.
Step 4
Monitor, evaluate, and tune
Once connected, use Orq’s dashboards to watch latency, errors, and cost for Haystack pipelines and agents, and run evals or experiments on their traces. You can adjust routes, models, or prompt configurations centrally where supported, while Haystack continues to handle components, pipelines, and agents
Use Cases
Multi‑agent products with real observability
Trace which Haystack agent, pipeline component (retriever, generator, ranker, tool), and model contributed to a failed workflow or unexpected outcome, combining Haystack’s own views with Orq’s router‑level traces.
Cost‑aware internal tools
Route routine steps like simple QA or summarization to lower‑cost models while keeping complex retrieval, multi‑hop reasoning, or enterprise QA on your strongest routes configured in Orq.
Eval‑driven agent improvements
Reuse failed conversations, pipeline outputs, and retrieval logs as eval inputs before shipping prompt, component, or model changes, turning production telemetry into a continuous improvement loop.
Safer experimentation across providers
Test a new model or provider on a small share of Haystack traffic, then promote or roll back based on traces and evals, instead of editing each pipeline definition or generator config by hand.
FAQ
Do I have to change my Haystack code to use Orq.ai?
In many cases, you can repoint the LLM generators Haystack uses (for example OpenAIChatGenerator) to Orq’s router by updating base URL and API key, keeping your existing pipeline and agent configuration as‑is. Components like retrievers, document stores, and tools stay in Haystack.
Does Orq.ai replace Haystack’s own runtime or framework?
No. Haystack remains your orchestration framework for pipelines and agents and can keep using OpenLIT, OpenTelemetry, or other observability tools. Orq.ai sits alongside it as the control plane for models, multi‑provider routing, cost tracking, and additional evaluation.github+1 You still design and run agents and pipelines with Haystack, but you use Orq to see what they cost, which models they should call, and how LLM behavior compares across providers under the same workflow.
Can I keep using my existing LLM providers with Haystack if I move to Orq.ai?
Yes. You can bring your existing provider keys (OpenAI, Azure, local models, etc.) into Orq.ai and route Haystack traffic through them, alongside any new models you add later, centralizing access and routing without losing your current integrations. Bring production controls to Haystack agents and pipelines
