# The data-quality problems that surface only in production

_Some data problems are invisible in a pilot by design. They live in the edge cases the demo never reached, and they wait until real users arrive._

The data-quality issues that never appear in a pilot and always appear in production: edge cases, conflicting sources, and stale records. Anonymized stories from a migration lead and a voice-AI founder.

# The data-quality problems that surface only in production

There is a category of data problem you cannot catch in a pilot, no matter how careful you are, because the pilot never runs long enough or wide enough to trigger it. Then you go live and it triggers on the first afternoon.

## The edge cases the demo never reached

An engineering lead who has moved close to three billion records between business systems described how he actually handles data transformation: cover the most common cases first, then work toward the edge cases until you are satisfied, and for the small ones, sometimes just fix the records by hand. That works because he knows the edge cases exist. The teams that get burned are the ones who saw the common cases pass in the pilot and assumed the tail did not matter. In production, the tail is where your loudest users live.

This is why so much of the industry reporting lands on the same finding: roughly 95% of pilots stall on integration and trust, not on model quality. The model handled the demo. It was the ragged real-world data, the records nobody curated, that broke the promise.

## Conflicts that only appear at volume

Two systems holding the same customer will eventually disagree. One allows duplicates, the other does not. One says the order shipped, the other says it is still open. In a pilot with a few hundred hand-picked records, these collisions are rare enough to miss. At production volume, with orders flowing across point of sale and e-commerce at the same time, they are constant, and each one is a customer whose order is stuck. The migration lead's rule of thumb, last write wins, exists precisely because someone learned this the expensive way.

## The live demo that failed

A founder who built a voice-AI product told a story every builder recognizes. The demo failed in front of a client. Not because the idea was wrong, but because live conditions are not rehearsal conditions. Real inputs arrive in an order you did not script, at a pace you did not test, in a shape your curated examples never contained. Data quality is the same. The clean sample behaves. The live stream does not.

## Stale data is its own failure mode

There is a subtler version that pilots almost never catch: data that was correct when the sample was taken and wrong by the time production runs. Addresses change, prices update, inventory moves, customers churn. A curated snapshot freezes the data at one clean moment, so the pilot never experiences freshness as a problem. Production experiences it constantly, because the world keeps moving while the model reads a record that stopped being true. A feature that looked accurate in the demo can drift into being confidently out of date, and nobody notices until the answers stop matching reality.

## How to drag these problems earlier

You cannot eliminate production-only problems, but you can pull most of them forward. Run the pilot on a genuinely random, genuinely messy sample instead of a curated one. Deliberately include stale records, conflicting sources, and the oldest data in the system. Log what the feature does with the inputs it handles badly, not just the ones it handles well. The goal is to meet the edge cases in a controlled week, not in front of a customer.

## How we approach it at Density Labs

In the AI Readiness Assessment, our fixed two week, $2,500 engagement, we go looking for the failure modes on purpose. We ask which records are stale, where two systems disagree, and what the feature does when it meets a record it was not designed for. Finding those in a diagnosis is cheap. Finding them after launch, when they are attached to real customers, is not.

A pilot proves the happy path works. Production is where you find out if you built anything else.