---
title: "Enterprise RAG: why it retrieves the wrong document"
source: "https://www.matteovincenti.com/en/blog/enterprise-rag-wrong-document"
language: en
author: "Matteo Vincenti"
published: 2026-10-24
category: "AI"
tags: ["RAG", "Document retrieval", "PostgreSQL"]
---

# Enterprise RAG: why it retrieves the wrong document

A reproducible lab compares full-text, embeddings and hybrid search to show when to fix missing sources, eligibility or ranking.

![A green ventilated device between two open manuals: one shows its switch, the other its open intake grille.](https://cdn.sanity.io/images/xrqkwry7/production/aeb95953c910571837a38ec7eccfa35d5201e5c0-1536x1024.png?w=1400&fit=max&auto=format)

Category: AI · Tags: RAG, Document retrieval, PostgreSQL

An enterprise assistant receives a question: “How do I stop AX-42 overheating without shutting the unit down?” Its first result describes an emergency shutdown when smoke appears. The procedure for reducing the workload and clearing the air intake comes second. Both documents concern the same product and temperature; they support different decisions.

This happens before the answer is written. Retrieval-Augmented Generation, or RAG, searches an external collection and supplies the retrieved material to a model that generates the response. If retrieval misses a necessary source, the generator receives incomplete evidence. If the source is present, how the generator uses it still needs checking. [Lewis and colleagues' RAG paper](https://arxiv.org/abs/2005.11401v4) combines generation with retrievable external memory.

Before changing the model, I would identify the evidence each question needs. In the synthetic lab accompanying this analysis, full-text search already retrieves every required piece of evidence for the seven answerable cases within the first three results. Semantic and hybrid retrieval change some positions without adding coverage. The missing-contract case fails with every method.

## What evidence must reach the answer?

A relevant document may cover only part of a decision. To establish whether customer North can request a replacement for an AX-42 bought twenty days ago, the assistant needs the policy's request window and North's contractual coverage. Retrieving the thirty-day policy alone leaves part of that decision unresolved.

The lab declares two evidence identifiers:

```json
{
  "query_id": "Q07",
  "required_evidence_ids": [
    "returns-v2#window",
    "north-contract-v1#coverage"
  ]
}
```

The part before `#` identifies a document revision; the part after it identifies a passage. This is the example's own format. Expected evidence is written before the comparison and read by the evaluator after retrieval. The rankers do not consult it to choose results or decide that they cannot answer.

The list makes a specific question testable: did both pieces of evidence arrive? Judging only the relevance of retrieved passages could leave the missing contract unnoticed. The report therefore distinguishes the number of required passages retrieved from the presence of the complete evidence set.

For an initial evaluation set, I would select requests for which the team can identify the necessary documents, alongside requests the collection cannot answer. I would include the short formulations and codes people actually use, as well as the well-formed questions from a prototype. The initial work is agreeing what counts as a supported answer.

## Which sources may this request use?

The collection contains AX-42 and AX-24, two revisions of the technical specification, an earlier and a current returns policy, and an operations-only document. Similarity to a question does not determine a document's effective date or the requester's permissions.

Every method applies the same checks before ranking:

- the product explicitly named in the question;
- published documents effective on the requested date;
- at least one authorized group shared with the fixture identity.

The earlier policy remains correct for a question about 1 February 2026. A filter admitting only the current revision would prevent that historical lookup. The system therefore needs a reference date and defined validity intervals; a file's upload date serves a different purpose.

The temporal and group selection in the code is:

```sql
WHERE d.product_id = %s
  AND d.status = 'published'
  AND d.allowed_groups && %s::text[]
  AND d.valid_from <= %s::date
  AND (d.valid_to IS NULL OR %s::date < d.valid_to)
```

Python passes the parameters to the database. Start dates are inclusive and end dates exclusive. Groups come from trusted test configuration; a real service must derive them from an authenticated identity and keep permission metadata synchronized. Microsoft's [security filter documentation](https://learn.microsoft.com/en-us/azure/search/search-security-trimming-for-azure-search) illustrates applying the filter to every query. Hiding the group field in the output does not enforce authorization.

Where filtering happens also matters. Retrieving five results from the whole archive and then discarding four can leave authorized documents outside the selected set. Microsoft documents this risk in its [vector filter modes](https://learn.microsoft.com/en-us/azure/search/vector-search-filters). The lab selects eligible sources before building each ranking and performs exact search over a small vector set.

The internal exceptions document is excluded for sales and admitted for operations. That is a result of the shared eligibility control. It does not show that a ranker understands permissions better. The same applies to AX-24, removed by the common product filter.

## Compare ranking on the same material

Each section becomes a *chunk*: an indexed passage. Search text includes the document title, section heading, and body. Its document reference preserves the revision: `rack-guide-v2#airflow` is never joined to an older guide. Every method receives these same fields.

PostgreSQL full-text search converts words into lexemes, applies English stemming and stop words, and orders matches with `ts_rank_cd`. The query combines lexemes with OR. Requiring every word in a natural-language question would have created an unnecessarily weak comparison. This is an explicit lexical baseline, distinct from BM25. [PostgreSQL's documentation](https://www.postgresql.org/docs/18/textsearch-controls.html) describes parsing, configuration and ranking, and explains that relevance depends on the application.

Semantic search, also called dense retrieval, computes an *embedding*, a numerical vector produced by a model from text. Queries and passages enter the same space, and the code ranks their cosine similarity. The encoder here is the public [all-MiniLM-L6-v2 model](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2): 384 dimensions, executed through its ONNX artifact on CPU. Tokenization, attention-mask mean pooling and normalization follow the model files. Inputs exceeding 256 wordpieces fail explicitly.

The third method combines the lexical and semantic rankings with *Reciprocal Rank Fusion*, or RRF. It sums contributions based on position, rather than directly adding scores with different scales:

```py
score[evidence_id] += 1 / (60 + position)
```

The constant is 60, the candidate pool is at most 20, and evaluation inspects the first three and five results. These values, the questions and the expected evidence were fixed before output. After the common product filter, the explicit product code is removed from the ranking query for every method: it has already selected the product.

## What changes across the ten cases?

The table counts required evidence retrieved within the first three results. Ranks are shown where they clarify the difference. A dash means the fixture contains no eligible supporting evidence for that request; recall and completeness remain null.

_What changes across the ten cases?_

| Case | Full-text | Dense | RRF |
| --- | --- | --- | --- |
| Q01 — AX-42 maximum temperature | 1/1, rank 1 | 1/1, rank 1 | 1/1, rank 1 |
| Q02 — cool AX-42 while keeping it running | 1/1, rank 2 | 1/1, rank 1 | 1/1, rank 1 |
| Q03 — current replacement request window | 1/1, rank 1 | 1/1, rank 1 | 1/1, rank 1 |
| Q04 — window on 1 February 2026 | 1/1, rank 2 | 1/1, rank 3 | 1/1, rank 3 |
| Q05 — internal exception requested by sales | — | — | — |
| Q06 — the same request from operations | 1/1, rank 1 | 1/1, rank 1 | 1/1, rank 1 |
| Q07 — policy and North contract | 2/2 | 2/2 | 2/2 |
| Q08 — North contract not ingested | 1/2 | 1/2 | 1/2 |
| Q09 — undocumented underwater pressure rating | — | — | — |
| Q10 — airflow for rack installation | 1/1, rank 1 | 1/1, rank 1 | 1/1, rank 1 |

Coverage is unchanged with five results. Every method retrieves all sources for the seven answerable cases; the contract remains missing in the incomplete-ingestion case.

The cooling question shows why order also deserves attention. Full-text search places the emergency smoke shutdown before the procedure that keeps the unit operating. Dense retrieval and RRF put the latter first. At k=3 the useful passage was already available; whether a generator uses it correctly remains a separate check.

The historical question moves in the opposite direction: the required policy drops from second to third. A semantic signal can improve one ordering and worsen another. For the requirement “all evidence within the first three results,” I would retain full-text search on this set and investigate question ambiguity before adding the encoder.

Q08 calls for a different intervention. The comparison repeats North's question after deliberately omitting the contract from ingestion. The current policy is found, but no method can retrieve a document absent from the index. The evaluation report identifies the missing evidence.

Q09 asks for an underwater pressure rating that no document states. Every method ranks a passage about a maintenance hose's supply pressure first. The number is close to the topic and does not answer the question. Returning neighbors does not establish that supporting evidence exists. [UAEval4RAG](https://arxiv.org/html/2412.12300v3) studies the need to evaluate unanswerable requests too; here the expected-evidence set reveals the missing support without becoming a runtime abstention rule.

No candidate or context violated the common product, group, publication or validity checks. Title and section context are always supplied, so this comparison does not measure a gain from adding parent context. [Anthropic's Contextual Retrieval research](https://www.anthropic.com/research/contextual-retrieval) explains the problem of passages losing useful references; its results concern a different technique and setup.

## Reproduce the comparison in Docker

The reproducible “rag-evidence-lab” package ([create\_lab.py](https://www.matteovincenti.com/downloads/rag-evidence-lab/)) provides one complete generator for all eighteen files. Save the page's Python block as `create_lab.py` in a new working location and run:

```bash
python3 create_lab.py
cd rag-evidence-lab
```

The script validates its payload and every file hash before creating the directory; it refuses an existing `rag-evidence-lab` directory. The individual sources on the page are available for inspection: the generator writes them all. The README and the commands below guide workspace preparation and execution.

You need a running ARM64 Linux Docker engine, the `docker compose` command, and an existing Python 3.10+ host interpreter for preparation and hashing. The recorded run used Engine 29.9.2 and Compose 5.6.1 on ARM64. The lock contains Linux ARM64 wheels. An AMD64 port needs a separate plan rather than an assumption that emulation has the same costs.

The runner is limited to two CPUs and 4 GiB RAM; PostgreSQL has 512 MiB. Reserve 5 GiB disk. The fixed plan contains 145,987,274 bytes of artifacts, about 139.2 MiB, in addition to base images. Corpus and queries are English in both article versions because the selected model is English; the comparison does not evaluate translated queries.

These files have distinct roles:

_Reproduce the comparison in Docker_

| File or group | What it supplies |
| --- | --- |
| `fixtures/documents.jsonl` | Documents, revisions, validity, groups, and sections |
| `fixtures/queries.jsonl` and `expected.jsonl` | Request inputs and separate expectations for the evaluator |
| `config.json`, `artifact-plan.json`, `requirements.lock` | Parameters, artifact URLs/sizes/hashes, and pinned CPU dependencies |
| `compose.yaml`, `Dockerfile`, `versions.env.example` | Services, internal network, and pinned images |
| `prepare_workspace.py`, `download_model.py` | Work directory preparation and verified downloads |
| `encoder.py`, `schema.sql`, `lab.py`, `retrieval.py`, `evaluate.py` | Encoding, indexing, retrieval, and post-hoc evaluation |

Choose a new work path. Preparation refuses an existing directory:

```bash
export RAG_WORK=/tmp/rag-evidence-lab-run
python3 prepare_workspace.py "$RAG_WORK"
docker version
docker compose version
docker compose --env-file "$RAG_WORK/versions.env" -p rag-lab config --quiet
docker compose --env-file "$RAG_WORK/versions.env" -p rag-lab run --rm model-download
```

The downloader verifies size and digests, then creates the wheel directory, model cache and download receipt in the workspace. It uses pinned public URLs without a token. Do not run `metadata_plan.py` to reproduce this run: it is the optional tool for preparing a new plan and would resolve different current versions.

Build the runner offline, then start the database:

```bash
DOCKER_BUILDKIT=0 docker build --rm --force-rm --no-cache --network none \
  --build-arg PYTHON_IMAGE=python@sha256:c3729439afc4ec3f3e159e0c86aeabfadac8030ef46838a0256383d0511c5f5e \
  -t rag-evidence-lab:20261024-onnx-cpu "$RAG_WORK"
docker compose --env-file "$RAG_WORK/versions.env" -p rag-lab up -d --wait db
python3 lab.py freeze
docker compose --env-file "$RAG_WORK/versions.env" -p rag-lab run --rm runner python lab.py run
docker compose --env-file "$RAG_WORK/versions.env" -p rag-lab run --rm runner python evaluate.py
```

Disabling BuildKit selects the legacy builder used for this package and avoids leaving task layers in a shared build cache. A deprecation notice may appear. If the engine no longer supports that builder, define a different build and cleanup path before executing the experiment.

The download service has external network access. The runner and database communicate only over the internal Compose network, with no host ports. No host packages are installed. Answer generation is outside this flow.

The `freeze` command records hashes of the code, configuration and fixtures. The run rejects a changed freeze or an existing result. If a fix requires another attempt, retain the previous receipt before preparing a new workspace.

The resulting files are:

- `output/retrieval.json`: ordered results, contexts, query lexemes, and environment;
- `output/evaluation.json`: required, retrieved, missing, and forbidden evidence for each case and k;
- `output/document-embeddings.npy`: vectors actually produced by the model.

Inspect the per-case table and the absence of forbidden candidates. Cases without expected supporting evidence retain null values; incomplete cases preserve the missing identifier. A single “test passed” would erase those distinctions.

Retain what is needed for comparison before cleanup. Then remove this project's services and networks:

```bash
docker compose --env-file "$RAG_WORK/versions.env" -p rag-lab down --volumes --remove-orphans
docker image rm rag-evidence-lab:20261024-onnx-cpu
```

Remove the selected workspace after saving the receipts. Base images may be shared with other work, so check their use before removing them. The package requires no global prune.

## Decide what to fix

The team's first question is whether the required source existed, had been indexed, and was eligible for the request. A missing contract calls for an ingestion fix. A document from the wrong revision or group calls for an eligibility fix. When valid evidence stays outside useful results, comparing queries, chunking and ranking is appropriate.

Lexical and semantic retrieval provide different signals. Codes can benefit from exact fields, while a paraphrase can benefit from embeddings. What matters is which questions change and which requirement improves. Hybrid retrieval adds no evidence within k=3 in this comparison. Introducing it simply because it is available would add integration work without an observed coverage benefit.

If some information lives in a [context wiki maintained by the team](https://www.matteovincenti.com/en/blog/context-engineering-memory-handoffs-wiki), retain the source link and assign responsibility for updates. Subsequent retrieval needs to distinguish that summary from the document it is based on.

A small, stable eligible corpus can also be supplied to the model in full, with cached context reuse considered where suitable. The [Cache-Augmented Generation paper](https://arxiv.org/html/2412.15605v2) explores that option for manageable collections. Questions depending on explicit relationships can justify comparing a graph or additional retrieval steps. None of these choices reconstructs a source that was never ingested.

Before changing architecture, ask for a readable record: the question, required evidence, eligible sources, and passages actually retrieved. It makes the next intervention specific enough to distinguish document work, eligibility controls, and ranking.