Deduplication before AI, or garbage at scale

An AI feature built on duplicated data does not fail loudly. It succeeds at producing wrong answers, confidently, at the speed of a computer. Every duplicate you leave in the pipeline gets multiplied by whatever the model does next.

What a duplicate actually costs

Start with a metric. Say a dashboard reports average revenue per customer, and the model uses that number to decide who gets a retention offer. Now suppose 8% of customers exist twice in the data because two source systems captured them separately. The count of customers is inflated, the revenue is split across phantom accounts, and the average the model reasons over is wrong before any AI touches it. The model is not making an error. It is faithfully computing on a population that does not exist.

An engineering lead who has moved close to three billion records between business systems sees this on every migration. One system lets you create duplicate customers, the next one does not. So the same person shows up twice in the source and has to become one record in the destination, and something must decide that they are the same and merge them. When that decision does not happen, the duplicate flows downstream and quietly doubles its own weight in every calculation.

Duplicates hit AI in three places at once

Metrics are only the first surface. Feed duplicated records into a training set and you have told the model that some examples matter twice as much as they should, tilting whatever it learns toward the accounts that happen to be duplicated. Feed duplicates into a retrieval system and it returns the same fact two or three times, crowding out other context and making the model more confident in whatever got repeated. The duplicate does not sit still. It compounds through every stage you built.

AI features generally need something on the order of 85% accuracy to hold up, and below that line errors start to compound. Duplication is a direct route under that line, because it corrupts the inputs, the training signal, and the retrieval context all at once, without ever raising an error.

Why the pilot never sees it

Dedup gets skipped because the pilot is run on a curated sample that a human already cleaned. Someone pulled a few hundred tidy records for the demo, the obvious duplicates were removed by hand, and the model looked great. Production does not come pre-cleaned. It arrives with every duplicate the source systems ever created, and the feature that shone on the sample starts double-counting on day one.

The migration lead’s method is the tell. She phases every large migration, runs the core records first, and has a human reconcile them, merging the duplicates, before she reruns the tool for the incremental data. That reconciliation is not glamorous and it is not fast. It is the step that keeps the destination from inheriting the source’s duplicates, and it exists precisely because no one can dedup three billion records by faith.

Dedup is a decision, not a checkbox

Deduplication is not a switch you flip. It is a set of choices. What makes two records the same, how close is close enough, and what happens to the fields that disagree once you merge them. Those are judgment calls someone has to own before the model is trained, not questions the model can answer for you. Treat dedup as required prep and the model gets a clean base. Treat it as something to handle later and later becomes production, where the garbage is already at scale.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two week, $2,500 engagement, we look for duplication before we look at the model. We check whether the same entity can exist more than once across the sources a feature uses, how duplicates are detected, and who owns the merge rule. Where duplicates flow through unreconciled, we flag it, because a duplicate that survives into training or retrieval is an error the model will happily amplify. Cleaning it in week one is cheap. Finding it in a stalled pilot is not.

Duplicated data does not slow AI down. It speeds up the production of wrong answers.