The regression that only shows up at the tenth percentile

A change ships. The average evaluation score ticks up. The team celebrates and moves on. Three weeks later the support queue fills with a specific kind of complaint, and when someone finally digs in, the change that raised the average also made the worst cases worse. The mean went up because the easy cases improved. The hard cases quietly fell off a cliff, and the average was never built to warn you.

Averages are built to hide the tail

An engineering leader who scaled large consumer systems said his teams stopped trusting the mean years ago. A single average rewards a change that helps the many and hurts the few, and for a lot of products the few are exactly the cases that generate the anger. He watched the tenth percentile, the worst tenth of outputs, as closely as the average. A change was only good if it lifted the floor or at least did not lower it.

This is the shape of the trap. Optimization pressure flows toward the common cases because that is where the volume is, and the metric you chose decides whether the rare cases are protected or sacrificed. Pick the mean, and you have quietly told the system that the tail is expendable.

The tail is where the cost lives

A VP of engineering at a mid-market logistics company saw it in production. Her team tuned a routing-summary feature and the aggregate quality rose. But the feature had gotten worse at the unusual shipments, the exceptions, and those were the only cases anyone needed the feature for in the first place. The common shipments were fine without any AI. The change looked like progress and was a regression where it counted.

The average had every right to look good. It was averaging over a pile of easy shipments the feature barely needed to touch, and a handful of hard ones where it now failed. She started reporting quality on the exception cases separately from the whole. The number was lower and less flattering, and it was the one that actually predicted whether the feature was earning its place.

How to watch the tail, not just the mean

Report percentiles, not one average, and treat the worst tenth as a first-class number.
Slice by case type, so the rare-but-important inputs get their own score instead of drowning in volume.
Gate changes on the floor, refusing a change that lifts the mean while lowering the worst cases.
Keep the hard cases in the set, because a tail you do not measure is a tail you are optimizing against by default.

The number that catches it

A single score per release is a summary of the cases you were never worried about. The number that catches a tail regression is boring to compute and easy to skip: the score on your worst slice, tracked release over release, held to its own bar. When it drops, you have found the regression before the queue did. When it holds, you know the change was safe for the people it was most likely to hurt.

The average is a friendly number that agrees with you. The tenth percentile is the one that argues, and it is the one worth reading first.