Training-serving skew: the same feature computed two different ways

Here is a failure that hides in plain sight. A model is trained on a feature computed in one place, then served a version of the same feature computed somewhere else, by different code, and the two do not agree. The model learned on one number and now makes decisions on another.

Two pieces of code, one feature

A feature is rarely a raw field. It is usually a calculation. Days since the customer’s last order. Average basket value over ninety days. Whether an account has ever triggered a refund. Someone writes that calculation once, in an offline job over the historical warehouse, to build the training set. Then someone else, often a different engineer on a different deadline, writes it again in the live service so it can be computed at request time. Now the same feature has two implementations.

They start out meant to be identical. They drift apart in the details. The offline job counts calendar days, the live one counts elapsed time and rounds differently. The offline version excludes cancelled orders, the live version forgot to. A null in the warehouse becomes a zero in production. None of these look like bugs. Each one shifts the number the model receives at inference away from the number it was trained to trust.

A pipeline PM saw the shape of this

A product manager whose whole job is moving data out of fragmented SaaS apps into a warehouse described the root of it without naming the term. Her connectors pull the same customer from five systems, and each system captured that customer differently because their form validations differ. The “same” field is not the same across sources. Now imagine building a feature offline from the warehouse copy and serving it live from one source system’s copy. You have two truths for one number before the model is even involved. That is training-serving skew waiting to happen.

The model cannot tell you this is wrong. It receives a value, it produces an answer. If the value is subtly off from what it learned on, the answer is subtly off too, and the feature underperforms in production while acing every offline test. AI features generally need on the order of 85% accuracy to hold up, and skew is a quiet way to sit just under that line without any red light explaining why.

The tell is a gap between offline and online scores

Skew has a signature. The model looks strong when you evaluate it on the offline data, and weaker when you measure it on live traffic, and no one can explain the gap because the model is the same model. Teams often chase this by retuning the model, which does nothing, because the model was never the problem. The problem is that the two feature pipelines disagree, and the offline evaluation and the online reality are computing from different definitions.

One source of truth for features

The fix is structural. Compute a feature once, in one place, and use that same computation for both training and serving. When the definition of “days since last order” lives in a single function or a shared feature store, the training set and the live request draw from the same logic by construction. There is no second implementation to drift. When you change the definition, you change it once, and both sides move together.

That is more upfront work than writing the calculation twice, and that is exactly why teams skip it under a pilot deadline. The pilot computes features in a notebook and serves them from a quick production script, the two happen to line up on the clean sample, and the skew only surfaces later, at scale, on the messy inputs the notebook never handled. By then the feature is live and the debugging is archaeology across two codebases.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two week, $2,500 engagement, we ask a plain question about every feature a model depends on: is it computed once and shared, or computed twice and hoped to match. Where a feature has two implementations, one for training and one for serving, we flag it, because that seam is where offline scores and production reality quietly part ways. Finding it in week one is a design conversation. Finding it in month four is a rewrite.

If the model learned from one number, serve it that same number. Not its cousin.