Cold starts: the first request to your AI feature after a quiet hour

A backend lead at a logistics company got a complaint that the AI feature was slow, but only sometimes, and only for some people. Most requests were quick. A handful took many times longer, and the people who hit those were, as far as anyone could tell, unlucky. The team could not reproduce it on demand. When they tested, it was always fast, because they tested during the day when the feature was busy.

The slow requests were cold starts. The inference ran on infrastructure that scaled to zero when idle, which was good for the bill. But the first request after a quiet stretch had to wait for the whole thing to wake up: spin up the compute, load the model into memory, warm the runtime. That work took real time, and it landed entirely on whoever showed up first. During busy hours the feature never went cold, so it never happened. Early in the morning, after lunch, in a quiet region, the first user waited while the system woke up on their request.

The team optimized for the average and shipped the tail to the customer. The average was fine. The person on the edge of the idle window got a different product.

Idle is a decision, and someone pays for it at the door

Scaling to zero is a real saving and often the right call. The cost of it is not hidden, but it is easy to overlook: the first request after idle is slow because it pays for the wake-up that everyone after it skips. That request is not an outlier to explain away. It is a designed-in property of a system that turns itself off to save money. If the feature sits on a latency-sensitive path, where a slow response means an abandoned action or a visibly stuck screen, the cold start is a product decision made by accident.

It hides from the people who could fix it because they never feel it. Engineers test during active development, when the path is warm. Load tests keep it warm by definition. The cold start only shows up in the quiet hours, for real users, in the exact conditions the team is least likely to be watching. So it reads as random slowness, and random slowness is easy to file under “look into it later.”

What good teams do

Decide which paths are latency-sensitive and treat cold starts on those paths as a real cost, not an edge case.
Keep a warm pool or a keep-warm ping for the paths where the first slow request actually hurts.
Measure the tail, not the average. The cold start lives at the top of the distribution, where an average hides it.
Test after idle on purpose. Let the feature go quiet, then send the first request and time it.
Trade the savings deliberately. Scaling to zero on a batch job is fine; on a user-facing prompt it may not be.

How we approach it at Density Labs

Our AI Opportunity Assessment is a fixed two-week engagement at $2,500 that scopes real production work before you build. Cold-start behavior is one of the things we look at directly, because it is invisible in a demo and loud in production. We ask which paths a user waits on, whether the inference behind them scales to zero, and what the first request after a quiet window actually costs. Then we measure that request instead of the busy-hour average everyone quotes.

The finding tends to surprise people. The feature is fast in every test and slow for the customer who arrives first after a lull, and no one felt it because no one tested cold.

Scaling to zero is a fine choice as long as you know who pays for it. Keep the paths that users wait on warm, and measure the first request, not the hundredth.