How fast does the answer need to be, and who decided that

A platform lead at a mobile gaming studio told me about a feature that got built for a speed nobody needed. The idea was to use AI to recommend which in-game offer to show a player. The team assumed the recommendation had to happen in the instant the player opened the store, so they designed the whole thing around sub-second responses. Real-time serving, low-latency infrastructure, the expensive path. It worked. Then someone asked a question that should have come first. Does it actually have to be instant?

It did not. The offers refreshed once a day. A recommendation computed overnight would have been just as good to the player and a fraction of the cost to run. The team had spent its budget buying speed the problem never asked for.

Latency is a decision, not a default

His lesson was that the latency budget is a scoping input, and if you do not set it on purpose, the design sets it for you, usually toward “as fast as possible.” That default is one of the most expensive assumptions in AI work. The gap between an answer that has to arrive in a moment and one that can arrive in an hour is enormous. It changes the model you can use, the infrastructure you have to run, and the price of every answer.

He described the two worlds his team lives between.

Interactive. The answer has to arrive while a person or a game loop is waiting. That constrains everything. You need fast models, warm infrastructure, and tight engineering, and you pay for all of it continuously.
Batch. The answer can be computed ahead of time and looked up when needed. Now you can use a slower, better, cheaper approach, run it on a schedule, and store the result. Most of the constraints relax.

The interesting part is how often people assume interactive when batch would do. A recommendation, a risk score, a content tag, a segment. These feel like they need to be live because the player experiences them live. But the input often changes slowly, which means the answer can be prepared in advance. Live consumption does not require live computation. Confusing the two is how you end up paying real-time prices for a daily problem.

Ask who set the budget, and why

His habit now is to treat any latency requirement as a claim that needs an owner. When someone says a feature has to be real time, he asks who decided that and what breaks if it is a minute slower, or an hour. Often the answer is that nobody decided it. It was assumed, because instant felt better, and better felt free. It is not free.

A friend who runs data at a retailer told me she asks the question in money. She makes the team estimate what the interactive version costs to run versus the batch version, and then asks the business owner whether the speed is worth the gap. Sometimes it clearly is. A fraud check at the moment of payment has to be live, and the cost is justified. Sometimes the owner looks at the two numbers and picks batch without hesitation, because the freshness was never worth what it cost.

The point is that the choice gets made deliberately, by the person who owns the tradeoff, in discovery, before the architecture is poured. Latency decided late is latency decided by accident, and undoing it usually means rebuilding the part you spent the most on.

How we approach it at Density Labs

In the AI Opportunity Assessment, our two-week fixed engagement at $2,500, we make the latency budget an explicit input, not a leftover. We ask how fast the answer truly has to arrive, who is waiting for it, and how often the underlying inputs actually change. That last question is the one that reframes the most projects, because it reveals how many “real-time” features are really daily features wearing a costume. Getting this right in discovery often makes the whole build cheaper and simpler than the first sketch.

Before you design for speed, find out who decided the answer had to be fast. Half the time the honest answer is that nobody did, and the problem was patient all along.