Data leakage: when your test set already knows the answer

A team showed me a model that scored 96% on their evaluation set. In production it landed at 71%. Nobody could explain the 25-point drop, so they assumed production was noisy and kept tuning. The real cause was simpler and worse. Their test set was not a test. The model had already seen the answers.

An exam where the student saw the questions

Data leakage is when information from your evaluation set reaches the model before the evaluation. The model does not have to be good. It just has to remember. Grade it on questions it already answered and it aces them, which tells you nothing about how it handles a question it has never met. The 96% is a memory score wearing an accuracy label.

The trap is that leakage produces exactly the signal everyone wants to see. High numbers, a clean demo, a confident sign-off. The failure hides inside the success, which is why teams chase it in the wrong direction for weeks. They trust the 96% because it is high, and the height is the symptom.

The ways it sneaks in

Leakage rarely looks like cheating. It looks like an ordinary pipeline with one quiet overlap.

You dedupe after splitting instead of before, so near-identical records land in both training and test.
You compute a normalization or a feature over the full dataset, then split, so the test rows carry a fingerprint of themselves.
Your retrieval system indexes documents that include the exact answers your eval asks about, so the model reads the key at inference time.
The same customer appears in training and test under two record shapes, one system allowing duplicates and the next not, so the model recognizes the person rather than solving the task.

That last one is the kind of conflict a migration lead who has moved close to three billion records sees constantly. When one source system permits duplicate customers and the destination does not, the same entity exists twice, and a naive split scatters those twins across train and test. The model learns the customer, not the pattern, and your metric rewards it for the wrong thing.

Why the demo dazzles and production sags

Production never leaks to you in advance. Real inputs arrive genuinely unseen, so the model finally gets the honest exam it dodged during evaluation. That is the entire story of the 96% that becomes 71%. Nothing broke between demo and launch. The demo was measuring memory and the launch is measuring skill, and those were never the same quantity. The disappointment was baked in the moment the test set stopped being unseen.

And this runs straight into the reliability threshold that matters. AI features generally need about 85% or better real accuracy before errors stop compounding. A leaked benchmark can show you 96% while your true number sits under that line, so you ship something you believe is safe and it degrades in front of users. The inflated score did not just mislead you. It hid that you were below the bar.

Catch it before it flatters you

The checks are cheap and boring. Split first, then dedupe, and check for near-duplicates across the boundary, not just exact matches. Group by entity, so all records for one customer stay on one side of the split. Confirm your retrieval index cannot serve a document that contains the eval answer. When a score looks too good, treat that as a symptom to investigate, not a win to celebrate. A clean test set often scores lower, and a lower honest number is worth more than a high leaked one.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two week, $2,500 engagement, a suspiciously high score is something we probe rather than trust. We check how the split was made, whether entities span both sides, and whether retrieval can leak the answer into context. Finding the leak in week one is far cheaper than discovering it as a 25-point drop in month three of a stalled pilot.

If the test already knows the answer, it is not testing anything. Make sure the exam is unseen.