The p99 latency problem nobody scopes in an AI pilot

A product director at a logistics company told me their new AI feature was performing well. The dashboard said the average response time was comfortable. The team was proud. Then he mentioned, almost as an aside, that a few large customers kept saying the feature felt broken. Same feature, same code, same dashboard showing green. The two facts sat next to each other and nobody had connected them.

They were both true. The average was fine. The tail was not.

Averages hide the customers who matter most

An average latency tells you what a typical request felt like. It says nothing about the worst requests, and the worst requests are where your reputation lives. If most calls return quickly but one in a hundred takes many seconds, the average barely moves. The dashboard stays green. But that one-in-a-hundred is not spread evenly across happy strangers. It clusters on your heaviest users, the ones who make the most requests, because the more you use a feature the more often you draw the slow tail.

Model calls have fat tails by nature. Provider load varies. A longer input takes longer to process. A retry after a timeout stacks one slow call on top of another. Occasional queueing on the provider side adds seconds you did not budget for. None of this shows up in an average. All of it shows up in your p99, the latency that one percent of requests exceed.

Here is the uncomfortable math. A customer who makes a hundred requests in a session will almost certainly hit your p99 at least once. Your best customers hit it constantly. So the number that predicts whether your loudest accounts stay happy is not the average. It is the tail.

Set a tail budget, not an average

The fix is to change what you promise. Do not scope the feature around “average response time.” Scope it around a tail-latency budget: p95 and p99 targets that the feature must stay under. Then measure against those targets, alert on those targets, and make engineering decisions against those targets.

Teams that get this right tend to do a handful of plain things.

They report p95 and p99 next to the average, never the average alone.
They set an explicit tail budget before launch and treat a breach as a real defect, not a curiosity.
They add a timeout that caps the worst case, so a stalled model call cannot drag the tail to infinity.
They design a fallback for the slow path, so a request that blows the budget still returns something useful instead of hanging.
They watch the tail on their heaviest accounts specifically, because that is where the tail concentrates.

A tail budget changes the conversation from “it feels fast” to “it stays under this number for ninety-nine of every hundred requests.” That second sentence is one you can defend.

How we approach it at Density Labs

Most AI pilots that disappoint do not disappoint on quality. They disappoint because the team measured the middle and the customers felt the edge. The average was a comfortable story that the loudest accounts never lived in.

Our AI Opportunity Assessment is a fixed two-week engagement, priced at $2,500, that scopes the real production work before you build. Part of that is deciding, at design time, what the tail-latency budget should be and how the feature behaves when a call exceeds it. That is a cheaper conversation to have before launch than after a major customer has decided the feature is broken.

Set a number your worst request has to beat. The average will take care of itself.