Entity resolution is the unglamorous work AI keeps tripping over

Two records land in front of you. Same name, slightly different address, one has a phone number and the other has an email. Are they one customer or two? That question has no clean answer, it comes up millions of times, and the AI you are building assumes it was already solved.

The problem hiding under every dataset

An engineering lead who has moved close to three billion records between business systems runs into this on every project, and she does not treat it as a footnote. She treats it as the work. In one of her migrations, the same person exists in five systems, and each system captured that person differently because their form validations differ. One required a last name, one did not. One stored the phone with dashes, one without. One let the customer enter a nickname. Five records, one human, and no field that agrees across all five.

Deciding those five are the same person is entity resolution. It sounds like plumbing. It is one of the hardest calls in the whole pipeline, because there is no universal key that ties the records together, only a pile of imperfect signals and a threshold someone has to set. Match too loosely and you merge two different people. Match too strictly and one customer becomes three in your data.

AI does not solve this, it inherits it

Here is the trap teams walk into. They assume the model will sort the mess out. It will not. If the pipeline hands the model three records for one customer, the model treats them as three customers. It counts their orders three times, splits their history into thirds, and reasons over a person who does not exist. A retrieval system pulls one of the three fragments and misses the other two. A training set built on unresolved entities learns from a population that is partly fictional.

The model has no way to know that “J. Smith” in the CRM and “John Smith” in the billing system are the same account. That knowledge is not in the data. It has to be constructed, by a resolution step, before the model ever runs. Skip the step and every downstream number is quietly wrong. AI features generally need something on the order of 85% accuracy to hold up, and duplicate entities can eat that margin before the model makes a single prediction.

Why nobody budgets for it

Entity resolution loses every funding argument because it produces nothing anyone can demo. The slide says “AI-powered customer insights.” Nobody writes a slide that says “we spent three weeks deciding which records are the same customer.” So the resolution work gets waved off as something the team will “handle in the pipeline,” and the pilot runs on a hand-cleaned sample where the duplicates were already reconciled by a person, quietly, before the demo. Then production arrives with its five-systems-one-customer reality and the feature that looked sharp on clean data starts double-counting.

The migration lead’s own habit shows how much of this is deliberate. She phases every large migration, runs the core entities first, and lets a human reconcile them before she reruns the tool for the incremental records. That reconciliation exists because no automated match is perfect and someone accountable has to make the ambiguous calls. It is slow, it is unglamorous, and it is the reason her migrations hold up.

The step you cannot skip

Resolving entities is not optional prep you can defer. It is the foundation the model stands on. A team that treats it as a first-class step, with a defined match logic, a threshold someone owns, and a human in the loop for the ambiguous cases, gives the model a clean population to reason over. A team that treats it as an afterthought gives the model a hall of mirrors and blames the model when the answers come back strange.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two week, $2,500 engagement, we ask how the same entity is identified across every system a feature touches. Where a customer or product can exist under different keys in different sources, we flag it, because that is exactly where an unresolved duplicate becomes a wrong answer the model cannot catch. We would rather map the five-systems-one-customer problem in week one than watch it corrupt a year of production data.

The model can only be as real as the entities you hand it.