Your pilot data isn’t your production data, and why that kills projects

Here is the quiet killer of AI pilots, and almost nobody names it out loud. The pilot ran on a clean, curated subset of the data. Production does not. Someone picked a few hundred tidy records for the demo, the model looked great, everyone approved, and then the real data started flowing and the whole thing wobbled.

Curated slices lie

An engineering lead who has moved close to three billion records between business systems described the shape of real data better than any textbook. Pull a customer export from one system and the shipping state reads “California.” The system you are loading into wants “CA.” That is one field, in one record type. Multiply it by every field, every system, and every merchant who entered data their own way for years, and the “clean” pilot slice starts to look like a fantasy.

His product catalog example is the same story. A demo shows the model reading “Nike red shirt medium” and correctly splitting it into product, color, and size. Then production hands it fifty thousand SKUs from an apparel store where the variants were entered inconsistently across a decade, and the neat little parser meets reality.

Two systems, two truths

The worst gaps are not typos, they are conflicts. In his migrations, one system lets you create duplicate customers and the next one does not. So the same person exists twice in the source and once in the destination, and something has to decide which record wins. His platform defaults to last write wins, but the point is that a human had to design that rule. In a pilot, that conflict never shows up, because the curated slice was deduplicated by hand. In production it shows up on day one, and if you never designed for it, a customer’s order silently breaks. In a restaurant integration, that means an order that never reaches the kitchen.

This is the killer the stat banks point at directly: pilot data is a clean curated subset, production data is not. It is also why so many pilots that “worked” cannot be scaled. The demo never touched the messy 20% where the actual engineering lives.

The gap is invisible until it is a customer

What makes this so dangerous is the timing. The conflict does not announce itself during the build. It waits until a real person places a real order, and then it appears as a support ticket, a missing shipment, or a kitchen that never got the ticket. By then the pilot has already been called a success and the budget has moved on. The migration lead’s own habit tells you how much of this work is deliberate: he phases every large migration, runs the core data first, lets a human reconcile it, and only then reruns the tool for the incremental records. That reconciliation step exists precisely because the real data never matches the pilot’s clean version.

The fix is to pilot on ugly data on purpose

The teams that avoid this do something slightly uncomfortable. They refuse to demo on the pretty slice. They pull a genuinely random sample, including the duplicates, the half-filled fields, and the records that predate the current schema, and they see how the feature behaves before anyone signs off. It makes the demo look worse. It also means the number you approve is the number you will actually get.

How we approach it at Density Labs

In the AI Readiness Assessment, our fixed two week, $2,500 engagement, one of the first things we ask for is a sample of your worst data, not your best. We look for the conflicts before you build: where the same entity lives in two systems, which field disagrees across sources, and what rule decides the winner. We would rather find the California-versus-CA problem in week one of a diagnosis than in month three of a stalled pilot.

A pilot that only ever saw clean data has not been tested. It has been flattered.