Why caching is the cheapest performance win an AI feature ignores

A platform team at an e-commerce company came to me wanting help picking a faster, cheaper model. Their AI feature felt slow and the bill was climbing, and they had convinced themselves the answer was a smaller model or a shorter prompt. Before we touched either, I asked to see the requests. That is usually where the real answer is, and it was here.

A large fraction of the traffic was repeats. The same product questions, phrased the same way, coming in over and over, each one running the full model from scratch as if it had never been seen. On top of the exact repeats there was a second layer: requests that were not identical but meant the same thing, worded a little differently, that also re-ran the model to produce an answer the system had effectively generated already. Every one of those paid full latency and full cost for a result that existed moments earlier.

They were tuning the engine while ignoring the fact that they were driving the same route a thousand times a day and never keeping the directions.

Cache before you optimize anything else

Caching is the cheapest performance win in an AI feature and the one teams skip most, because it is less interesting than model selection. But the cheapest request is the one you never send. Before you shave tokens off a prompt or shop for a faster model, you should not be re-running the model on work you have already done. That is not an optimization, it is stopping an obvious waste.

There are two layers worth building, and they catch different things. An exact cache keys on the precise input and returns the stored answer when the same request comes back. It is simple, it is safe, and for repeated queries it is close to free to serve. It handles the identical repeats completely.

A semantic cache goes further. It embeds the request and looks for a previous one that is close enough in meaning, so a differently worded version of a question you have already answered can be served from the cache too. It needs a similarity threshold and some care, because too loose a match returns a subtly wrong answer, but tuned properly it catches the near-duplicates that an exact cache misses. For the e-commerce team, those two layers together covered a large share of their traffic.

What good teams do

Look at the request mix before optimizing anything. Repeats and near-duplicates are the first thing to find.
Add an exact cache first. It is simple, safe, and handles identical repeats for almost nothing.
Add a semantic cache for near-duplicates, with a similarity threshold tuned so it does not return a close-but-wrong answer.
Set sensible expiry, so cached answers do not go stale when the underlying data changes.
Measure hit rate. A cache nobody measures quietly stops working after a data or prompt change.
Only then look at model choice and prompt length. Optimize the requests you actually have to send.

Once the two caches were in, latency dropped for the cached share because those answers came back immediately, and the bill fell because the model was no longer being run on work it had already done. Then, and only then, did the question of a faster model become worth asking, on the much smaller volume of genuinely new requests that remained.

How we approach it at Density Labs

Caching gets skipped because it is unglamorous, and it is skipped right up until someone looks at the request mix and sees how much of it repeats. Our AI Opportunity Assessment is a fixed two-week engagement, priced at $2,500, where we scope the real production work before you build. We measure how much of your traffic is exact repeats and near-duplicates, and design the caching layers, exact and semantic, that fit your data and your freshness needs. You leave knowing what a cache would save you before you spend a week chasing a faster model.

The cheapest request is the one you never send. Cache the work you have already done before you optimize the work you still have to.