TL;DR: RAG retrieval works by embedding a query and matching it against your documents, so anything that changes the query can move which documents come back. Orq's gateway can redact personal identifiable information (PII) from a search query before the embedding model ever sees it, and that rewrite is exactly the kind of change that matters. For queries that search on semantic meaning, it is nearly free. For queries that search on a person, it knocks the right document out of the top result about 60% of the time. How much this costs you comes down to one number: the share of your traffic that searches for people.
The query that breaks
We shipped PII redaction on the gateway, and then someone asked the obvious question we didn't have an answer to: if it scrubs a search query before the embedding model ever sees it, does retrieval still work? We didn't know, so we measured it.
Here's the kind of query that drops:
Sanne de Vries at Meridiaan Zorg who will reply from s.devries@meridiaanzorg.nl
The gateway sits between your app and the embedding model, so that's not the string the embedder sees. It strips the person and the address first:
<PERSON_1> at Meridiaan Zorg who will reply from <EMAIL_ADDRESS_1>
Still readable. But you've just deleted the two things the user was actually searching on, and in our benchmark, that knocked the correct document out of the top result about 60% of the time for queries like this one.
The positive news is for non-PII queries. Run the same redaction on a query that searches on semantic meaning (e.g., "the onboarding packet for the new logistics hire") and retrieval barely flinches: a few points, real but minor. So the honest answer to "can I use PII redaction with RAG?" isn't a single number. It's a question back: what are your users searching on?
Why this question comes up
Orq's AI gateway can redact PII on outbound calls, including the call that turns a query into a vector. Turn it on, and the embedding model never sees raw names, emails, or account IDs. Good for compliance. The problem arises when we search the database using embeddings from redacted queries. Two combinations can make sense here.
Data and queries are both redacted. Semantic similarity works as expected, but you can never look for PII-related information.
Queries are redacted, data is plaintext. An asymmetrical situation where just online queries are redacted, and the data is plaintext. This could be to prevent employee queries from leaking PII, for example. It's unclear how search is impacted in this situation, which is exactly what we wanted to quantify.
The query loses its PII at the gateway; the index, built earlier from raw documents, keeps it. Retrieval matches a scrubbed query vector against documents that still contain the person, and that gap is what the rest of the post measures.
The setup
Corpus: 600 short business paragraphs, each about a named person and their PII: email, phone, employer, location, and account. Realistic prose where the PII is a minority of the text. Every person and company is unique, so each query has one unambiguous right answer. This is the raw index.
Queries: 1,200 generated, two per document, tagged by what they anchor on; 1,110 kept after the filter below. About half are content queries that target the document's meaning; about half are pii queries that key off the person or a person-linked identifier. Every query runs through the PII redaction service.
One filter we applied before benchmarking: We kept only those whose gold document already landed in MiniLM's top 5 on the raw query. That left us with 1,110. The point is to measure redaction's cost against queries that retrievably work to begin with, not against queries that were broken before the gateway touched them. It's not fully symmetric: the gate dropped content queries (12.3%) about four times as often as person queries (2.7%), so the surviving content set is skewed toward the easy ones and its post-redaction drop is a floor, not a mean.
Two situations: baseline (raw query) and redacted (query scrubbed by the gateway), both ranked against the raw corpus. Two embeddings: a local MiniLM and OpenAI's text-embedding-3-small, so the result doesn't ride on one model. The numbers below lead with OpenAI: our query-selection step used MiniLM, so OpenAI is the backend that never saw the selection and gives the cleaner read.
This is a deliberately PII-heavy design. Both the corpus and the queries are built around named people, so roughly half the traffic anchors on a person, far more than most real RAG workloads. That makes the redaction effect easy to isolate, but it also means the person-query numbers below are a worst case, not what an average workload sees.
One detail that turns out to be load-bearing: each document is one chunk, embedded whole. No splitting, ~60 words per doc. Chunk size sets how big a fraction of the text the PII is, and that fraction is what drives the whole result. Hold that thought.
The results
Split by what the query targets, the two halves look very different:
recall@1 before (teal) and after (orange) redaction. Person queries collapse ~0.59 on both backends, meaning queries barely move. Two caveats live in this chart: the pii baseline sits near the ceiling (~0.99), so it also has the most room to fall. Meaning queries here are only the ones that already worked before redaction (that filter cut 12% of them vs 3% of person queries), so this is the easy end of the set. Read their small drop as a best case, not an average.
We expected person-anchored queries to take a hit. The size of it still caught us off guard: they lose roughly 60% of their recall@1 on both backends, and MRR drops to about half of baseline. Meaning-anchored queries lose a few points, statistically real, but an order of magnitude smaller. And the person-anchored collapse is not a small-sample fluke we're overreading: McNemar's exact test gives p far below 1e-100 on both backends.
We use McNemar's test to assess if the impact is due to the data being PII or not. This is the right significance check here because both conditions run on the same queries. It ignores the queries that behaved the same way twice and looks only at the ones that flipped: hit-then-miss versus miss-then-hit. If redaction had no effect, those two flip directions should be roughly balanced; a small p-value means they are lopsided. For person queries, almost every flip goes one way, redaction turning a hit into a miss, which is why the p-value is so extreme. For content queries, the flips are only mildly imbalanced, so the effect is real but small.
Why two metrics? They answer different questions. recall@1 is all-or-nothing: it counts a query as a hit only if the correct document lands at rank 1, so it tells you how often the top result is right. MRR (mean reciprocal rank) is more forgiving: it scores each query as 1 / (rank of the gold document) and averages, so a document at rank 2 still earns 0.5, rank 4 earns 0.25, and rank 10 earns 0.1. Reading them together tells you not just whether the correct document left the top slot but how far it fell. The person-query numbers are the interesting case: recall@1 craters to ~0.40 while MRR holds at ~0.55. That gap means the correct document is usually still there, just demoted: it sits around rank 2 on average rather than vanishing down the list. So redaction knocks person queries off the top result, but not out of contention.
Widen the window, and most of the "lost" documents come back: recall@10 for person queries only falls to 0.83 (OpenAI) and 0.76 (MiniLM), against ~0.40 at recall@1. Of the ~60% that lost the top slot, well over half are still sitting in ranks 2–10. A pipeline that feeds its top 10 candidates into a reranker recovers most of what recall@1 alone counts as gone. Content queries barely move on any of the three (recall@10 0.99 → 0.99, MRR 0.78 → 0.73), consistent with their small recall@1 dip.
Where the correct document lands after redaction. The teal-to-orange split is the story: person queries lose the rank-1 slot (dark teal) but most of that loss lands in ranks 2–10 (light teal), not out of the top 10 (orange). Only 17% (OpenAI) and 24% (MiniLM) of person queries fall out of reranker range.
Two honest footnotes on that table. First, content queries start lower at baseline (0.64–0.66) than person queries, not because of redaction, but because a meaning-only query legitimately matches several documents, so the single "gold" doc isn't always rank 1. Second, pool the two halves together, and you get a single "recall@1 fell ~40%" headline, but that average is a lie of composition, as it blends a catastrophic drop with a mild one. Nobody's workload is 50% person-queries. Yours has its own mix, and the number will shift based on that.
Why the person queries fall apart
We can watch the mechanism directly. Measure the cosine between each query's raw embedding and its redacted embedding, a direct measure of how far redaction moves the vector:
backend | pii | content |
openai-3-small | 0.62 | 0.96 |
MiniLM | 0.61 | 0.96 |
Content queries barely move (0.96). Person queries move a long way (0.61). That 0.61 is where the 60% comes from: recall tracks the shift almost one for one. A person query is short and the person is most of it, so redaction rewrites the vector; a content query is longer, and the PII is incidental, so redaction barely touches it.
The two means in the table hide two very different distributions. Content queries pile up against 1.0 (redaction leaves them untouched), while person queries are dragged down across the whole 0.4–0.8 range:
How far redaction moves each query vector (cosine of raw vs redacted). Content stays pinned near 1.0; person queries are pulled down to 0.4–0.8. OpenAI backend; MiniLM is near-identical.
And the shift predicts the damage, query by query:
Each dot is one query. Left to right is how much of the query vector survived redaction (1.0 = untouched); top to bottom is rank delta, the gold document's redacted rank minus its baseline rank, capped at +20, so 0 means redaction cost it nothing. The far left is the degenerate case: a query that is nothing but a name (Brendan Gordon Ramsey-Cole → <PERSON_1>) keeps almost none of its vector, cosine 0.15, and its document falls from rank 1 to 553.
Our content queries were longer than our PII queries, so we also have to understand the impact of longer queries, and by proxy, the percentage of a query that is PII. That is the chart below:
Every query, plotted against how much of it redaction removed. PII fraction correlates −0.96 with the surviving similarity; query length only 0.69. Keep length fixed (9–13 words, n=424) and content queries still keep 0.96 against pii's 0.63. Dot size is query length: big and small dots interleave along the whole band, so length does not explain the spread.
That last point is the whole story in one line. There is no wall between two regimes, just one continuous axis: the fraction of the query that is PII. A content query sits at the low-movement end, where the person is incidental and redaction barely touches the vector. A person query sits at the far end, where the person is nearly the whole query, so redaction rewrites it. Same mechanism at both ends; only the PII fraction changes.
What this means for your pipeline
You can run query-side PII redaction with RAG. Whether it costs you comes down to one number you can estimate from your logs: the share of queries that anchor on a person.
A documentation search where users type topics and error messages: turn it on, you'll pay a few points at most.
A support or CRM tool where users search by customer name or account: expect a real hit on those queries (more than half of them dropping out of the top result) and plan for it.
And because chunk size governs how much of the text the PII represents, a pipeline built on larger chunks will feel less of this than our one-paragraph chunks did.
Limitations
We tested one direction only: a redacted query against a raw index, because that's what the gateway likely uses. We did not test rebuilding the index from redacted documents too. Placeholders are numbered per call, so a <PERSON_1> in a query needn't match a <PERSON_1> in a document; the direction there is genuinely unknown.
The content numbers are close to a best case, not an average. We kept only queries that already retrieved their own document at baseline, and that filter dropped content queries (12%) about four times as often as person queries (3%), skewing the surviving content set toward the easy ones. The same filter, scored on MiniLM, is why the MiniLM baseline looks perfect and why we lead with OpenAI. The person-query collapse doesn't lean on either.
Synthetic, single-domain corpus at a fixed one-paragraph chunk size, generated by an LLM (qwen3.6-35b) with unique names so every query has one right answer. The effect scales with chunk size, and these numbers are the small-chunk end. Real corpora may differ.
The bottom line
Query-side PII redaction is not a hard yes or no for RAG. It's a tax whose rate you can estimate before you turn it on: look at what fraction of a typical query your redaction service would remove, and read the cost off that. Near zero for topic search, most of the query for person search, and a smooth line in between with nothing surprising on it. Key files for a given query tend to drop in rank but are typically still high enough for the LLM to use for RAG or other purposes. We only see catastrophic failures when most or all of the query is PII.
What we can't tell you is your own mix. Pull a week of queries, count how many name a person, and you have the number this whole post reduces to.
PII redaction and detection are now live in the orq.ai gateway. If you're looking to get started for your AI usage, have a look at our PII redaction guide.
