Your eval suite can be green and production can still be getting worse.
A support agent passes 500 regression cases on Monday. A customer sends a terse message nobody thought to include in the test set. The agent retrieves last year’s policy, gives a confident answer, and closes the ticket.
Nothing crashes. The first obvious signal might be the customer reopening the case.
Offline evals did their job. They tested the failures you already knew to look for. Real traffic keeps supplying new ones. Odd phrasing and combinations of inputs your test set never covered.
Online evaluation gives you a way to catch those cases while the application is running.
The useful cases don’t end in a dashboard.
Review them, work out what broke, and keep the important ones for the next offline eval run.
Offline evals gate changes, online evals catch what real traffic reveals
Offline evals are where you test changes before users see them.
Change the system prompt and rerun the same groundedness cases. Swap models and check whether tool selection gets worse. Adjust retrieval and see how the new setup performs on queries where you already know what a good result looks like.
That repeatability is the point. You want the test conditions to stay still while one part of the application changes.
Production does the opposite.
Users bring shorthand, strange phrasing, and workflows nobody thought to put in the eval set.
You won’t be able to enumerate all of that before launch.
What ends up happening is online evaluation watches how the released system behaves against the traffic it actually receives.
The useful failures then travel back into development.
If a new customer phrasing causes the agent to choose the wrong tool, review the trace and keep the case.
The next model or prompt release now has to pass it alongside the failures you already knew about.
Offline evals test what you already understand, online evals find the cases you missed.
Sample production traffic according to risk, not convenience
Evaluating 1% of production traffic sounds reasonable until the failures you care about live in the other 99%.
Random sampling is still useful. It gives you a baseline for how the application behaves across ordinary traffic.
Keep it, but deliberately pull in requests that deserve more attention too.
Sample type | What it helps you catch | Example |
Random baseline | Changes across normal traffic | 1–5% of standard chat requests |
Stratified sample | Problems concentrated in one cohort | A new language, tenant, model, or feature |
Risk-based sample | Failures with more serious consequences | Payment actions, compliance queries, external tool calls |
Exception sample | Requests already showing signs of trouble | Thumbs-down, retries, fallbacks, high latency, or weak retrieval |
We think fixed sampling rates are a poor default.
Evaluation coverage should move with the risk and the rate of change in the application.
A new knowledge base goes live? Inspect more requests that retrieve from it. Rolling out a new prompt to 10% of traffic? Sample that cohort heavily enough to compare it with the old version.
Human review should follow the same logic. Spend expert time where automated checks are uncertain, the consequences are higher, or behavior has started to move.
Orq.ai exposes evaluation sampling directly in agent configuration. In the current API example, an evaluator can be attached with a defined sample rate:
"evaluators": [ } ] |
The 50% value in this API example isn’t a recommended universal rate.
The useful part is that evaluation coverage is configurable, so you can adjust it to the workload rather than evaluating every request indiscriminately.
Match each evaluator to the failure mode it can actually detect
An 8.2 “quality” score dropping to 7.8 gives an engineer almost nothing to debug.
Did groundedness get worse? Are answers less complete? Is the agent choosing the wrong tool?
A single score can hide several different failures that need completely different fixes.
Give each evaluator one job:
Groundedness: Are claims supported by the retrieved context or tool results?
Completeness: Did the response address the important parts of the request?
Tool selection: Did the agent choose the right tool?
Tool arguments: Did it pass the account ID, date, or other required values correctly?
Retrieval quality: Did retrieval return the information the model needed?
Policy compliance: Did the response follow the rules for this request?
Format validity: Can the next system actually consume the output?
Some of these don’t need an LLM.
If the output must match a JSON schema, validate the schema.
Check tool arguments in code. Use pattern matching for known sensitive-data formats and allowlists where the rule is explicit. Asking another model to judge something your application can verify directly adds cost and another source of uncertainty.
LLM judges are more useful when the decision depends on meaning.
Whether an answer is grounded in its evidence, complete enough for the task, or faithful to a policy written in natural language.
Even then, test the judge before trusting its scores. Compare its decisions with human-reviewed examples and look closely at the disagreements. A grader that consistently rewards longer answers or misses errors in a specialist domain can make a dashboard look precise while measuring the wrong thing.
Score quality dimensions separately before reducing them to pass or fail
A pass/fail result is useful when the application needs to make a decision. Keep the score behind it too.
Suppose a groundedness evaluator uses 0.80 as its threshold. Last week, most responses scored around 0.93.
After a retrieval change, they cluster around 0.82.
Almost everything still passes.
The binary metric looks healthy, despite the system moving much closer to the failure boundary. Tracking the underlying groundedness scores makes that drift visible before the pass rate collapses.
Keep different quality dimensions separate for the same reason.
A response can be well grounded and still leave half the request unanswered. Another can be complete but rely on evidence that doesn’t support the answer.
Averaging those results into one “quality” number hides which part of the system is getting worse.
Reduce them to pass or fail only when the workflow needs a clear action.
The dashboard should still preserve the individual scores.
That’s what lets engineers see whether groundedness, tool use, or another part of the workflow started moving first.
Run evaluations inline only when the result needs to change the response
Every evaluator you put in the response path adds work before the user gets an answer.
Sometimes that delay is worth paying for.
Imagine a claims assistant generates an answer from retrieved policy documents. Before showing it to the customer, a groundedness check finds that one of the claims is unsupported.
A groundedness guardrail needs access to both the generated response and the evidence it should agree with.
Orq.ai’s LLM Guardrail test payload exposes the output alongside retrieved context and an optional reference:
{ |
Your application could then regenerate the answer using only the retrieved evidence and send the case for review if it still fails.
The same applies when a check can stop a bad tool call, catch sensitive data before it leaves the application, or reject output that another system cannot safely process.
Most quality checks don’t need that privilege.
A completeness score for a support conversation or a tone evaluation for a writing assistant can usually run after the response has already been returned. You still get the data for monitoring and future eval cases without adding another model call to the critical path.
A simple rule works well: put an evaluator inline when you already know what the application should do when it fails.
Otherwise, run it asynchronously and use the result to improve the system.
Our bar for putting an evaluator in the critical path is that its failure needs to trigger a defined action. Otherwise you're paying latency and cost for a score the application can’t use yet.
Monitor trends and distribution shifts, not isolated bad scores
One bad score is a trace to inspect. A pattern of bad scores tells you something changed.
Users will always produce difficult inputs, and models will occasionally fail on them. Alert on every failed evaluator result and the queue quickly becomes noise.
Look for movement that persists.
Maybe groundedness falls after a retrieval release. Tool-selection errors start climbing only on a new model route. One tenant suddenly produces far more incomplete answers than the rest of your traffic.
Those patterns are more useful than the individual failures because they give you somewhere to start looking.
Then open the affected traces.
The dashboard tells you that Tenant B's completeness dropped 12 points. The traces tell you whether prompts were truncated or users started sending a new class of request.
The alert should point you toward a change worth investigating, not every imperfect answer the model produces.
Feed production failures back into offline evaluation datasets
A production failure should make the next release harder to break.
Preserve enough of the original request to reproduce the failure:
The de-identified input and relevant conversation context
The behavior you expect the next version to produce
The prompt, model route, and configuration in use
A clear label explaining what went wrong
Retrieved documents and tool results
The response that failed
Now that case can run alongside the rest of the offline suite whenever the prompt, retrieval setup, or agent logic changes.
But you shouldn’t keep every bad response you find. Ten near-identical examples of the same mistake add volume without teaching you much.
Keep cases that expose something new. A failure mode the suite missed, an important traffic segment it barely covers, or an edge where the application behaves differently than you expected.
Over time, production traffic stops being just something you monitor. It becomes one of the ways your eval suite gets better.
Orq.ai connects online evaluations to traces, routes, and application changes
A falling eval score tells you where to look. Orq.ai lets you open the requests behind it.
Suppose groundedness drops after a knowledge base update. You can move from the aggregate result to the affected traces and inspect what those requests retrieved.
The problem may have started well before the model wrote the response.
Maybe the new knowledge base returned weaker evidence. If retrieval is the likely cause, inspect what changed in search, reranking, thresholds, or chunking.
A fallback route may be producing shorter answers with missing context. An instruction change could be sending the agent to the wrong tool more often.
We think the real measure of an online evaluation system is how quickly a team can move from “quality changed” to “this prompt, route, retrieval step, or tool call caused it.
Connecting the evaluation result to the execution path gives engineers something they can debug.
The requests that failed and the application behavior around them.
Treat online evaluation as a continuous quality feedback loop
Online evaluation is most useful when a production failure changes what you test next.
Start with one workflow. Pick the failures you care about, evaluate enough real traffic to catch them, and make sure someone reviews the cases that deserve attention.
When something breaks, open the trace.
Work out what changed. Keep the important failure as an offline eval case before the next prompt, model, or retrieval change ships.
That loop gets stronger over time because production keeps showing you cases your original test set never knew about.
See how Orq.ai connects online evaluations to the traces behind production quality changes.
