Data drift: the model that was right in March and wrong in June

A model can be correct on the day you ship it and wrong three months later without a single line of its code changing. The model did not move. The world it reads did.

The inputs are a moving target

Every model learns the shape of the data it was trained on. The average order size, the mix of product categories, the phrasing customers use, the times of day traffic arrives. It bakes those patterns in and assumes they hold. For a while they do. Then they drift.

Picture a demand model for a retailer, trained on records from a calm stretch in late winter and validated in March. It scores well. Everyone signs off. By June the store has rotated its catalog for summer, a promotion changed how people buy, and a new product line the model never saw is now a tenth of the volume. The inputs the model sees in June do not look like the inputs it learned in March. The predictions slide, and the dashboards that were green at launch say nothing, because none of this is an error. It is drift.

Silence is the dangerous part

A broken pipeline throws an alert. Drift does not. The data keeps arriving in the right shape, the job keeps finishing, the model keeps returning confident answers. They are just a little more wrong each week. AI features generally need something on the order of 85% accuracy before they hold up, and below that line errors start to compound. Drift can walk you under that line one slow step at a time, and the first real signal is often a human noticing the output feels off, months after it started.

That delay is expensive. By the time someone questions the model, the drift has been shaping decisions for a full season. The team’s instinct is to blame the model and retrain it, which helps, but only until the next shift. Retraining without monitoring is mopping a floor while the tap runs.

You have to watch the inputs, not just the outputs

The teams that catch drift early stop treating the model as the only thing worth measuring. They measure the inputs. They take the distribution the model was trained on and compare it, on a schedule, against the distribution flowing in this week. When the average order value climbs, when a category that was 5% of volume becomes 20%, when a field starts arriving with values the training set never contained, that is a flag. It does not mean the model is broken yet. It means the ground under it is moving, and someone should look before the output does.

This is cheap to set up and easy to skip. A pilot runs for a few weeks on data that barely has time to drift, the numbers look stable, and nobody builds the monitoring because nothing seemed to need it. Then the feature goes to production and lives for a year, through every season the pilot never saw.

Drift is a question of when, not if

Some inputs drift slowly and some lurch. A pricing change, a new market, a viral product, a policy update, any of these can move the distribution overnight. The point is not to predict which one. The point is to assume drift will happen and to have the instrument that tells you it has, so the conversation is “the inputs shifted last Tuesday” instead of “the model has felt wrong for a while and we are not sure why.”

A model is a photograph of a moment in your data. The moment does not last.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two week, $2,500 engagement, we ask what the inputs to a feature look like today and how much they are expected to move over a year. We flag the fields most likely to drift, seasonal volume, category mix, pricing, anything tied to the calendar or the market, and we recommend the monitoring that compares live inputs against the training baseline. Catching drift is an instrument you install once, not a fire you fight forever.

The model that was right in March is not lying to you in June. It is answering an old question.