Duplicate documents inflate your retrieval and hide the right answer

Retrieval returns a fixed number of chunks. If five of those slots go to copies of the same document, you have one piece of information taking five seats, and the authoritative source that deserved a seat is sitting just below the cutoff where the model never sees it.

The migration lead who thinks in duplicates

An engineering lead who has moved close to three billion records between business systems spends his working life on duplication, because duplication is what real data is made of. One system lets you create the same customer twice. The next one does not. So the same person exists twice in the source and once in the destination, and something has to decide which record wins. His platform defaults to last write wins, but the point he made is that a human had to design that rule. The duplicates do not resolve themselves. Somebody has to say which copy is canonical and collapse the rest.

A RAG corpus has the identical problem and almost never gets the identical discipline. Documents get uploaded more than once. A file is copied into a new folder and both survive. A near-identical version is saved with one paragraph changed. Nobody designed a rule for which copy wins, so all of them stay in the index, and retrieval treats every copy as an independent, equally valid chunk.

How copies eat the top-k

Here is the mechanics of the damage. Retrieval ranks chunks by similarity to the query and hands the top k to the model. Say k is five. A question comes in whose best match is a document that exists in the corpus in four near-identical copies. Those four copies are all almost equally similar to the query, so they take four of the five slots. That leaves exactly one slot for everything else, including the one document that held the missing detail the answer actually needed. It lands at rank six. The model never sees it.

So the duplicates did not just waste space. They actively evicted the right answer. The model reads five chunks that are really one chunk repeated, gets a narrow and possibly incomplete view of the question, and writes a confident answer from it. Retrieval reported five strong matches, so every signal looked healthy. The problem is that five strong matches were one document wearing five name tags, and the source that would have completed the answer got crowded out by copies.

Confidence goes up as quality goes down

The cruel part is that duplication can make the system look more sure of itself. Several highly similar chunks agreeing with each other reads like strong evidence. It is not. It is one piece of evidence counted five times. A model handed five copies of the same claim treats it as heavily supported, when the truth is you gave it a sample size of one and lied about the count. Genuine corroboration from independent sources and accidental repetition of one source look the same to retrieval, and only one of them should raise confidence.

Dedup is a rule someone has to design

The fix borrows straight from the migration lead. Decide which copy is canonical and collapse the rest, the same way he had to design last write wins for duplicate customers. That means detecting near-duplicates at ingestion, the exact copies and the ones with a single paragraph changed alike, and keeping one authoritative version instead of every draft that ever landed. It means a rule for which version survives when two disagree. This is deliberate work with an owner, not a setting you toggle, and it is the difference between a top-k full of distinct sources and a top-k full of echoes.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two week, $2,500 engagement, we look at how much of your corpus is duplicates and near-duplicates, and what that does to the top results on real questions. We check whether the top-k is five different sources or the same source five times, and whether the right document is getting pushed past the cutoff by copies of another. When it is, we help define the dedup rule and the canonical-version logic, the same discipline a migration lead applies to duplicate records before they ever reach production.

If your top five results are one document five times, retrieval is not searching. It is echoing.