Why the AI feature works until the third-party rate limit resets

A backend lead at a logistics company told me their new AI feature was “flaky.” It summarized shipment records and worked all the time in demos, then failed in clusters during the day and recovered on its own an hour later. The pattern looked random until we lined the failures up against the clock. They came in bursts, near the top of the hour, and then stopped.

The model was not the problem. Before the model ran, the feature called an enrichment API to pull carrier and address details for each shipment. That API allowed a fixed number of calls per hour. During a busy stretch the feature burned through the hour’s allowance early, the API started rejecting calls, and the summaries failed because they had no data to summarize. When the window reset, the allowance came back and the feature healed. The bursts of failure were the shape of a rate limit, drawn against time.

They had budgeted the model’s rate limit carefully. The dependency sitting right next to the model, the one that actually fed it, nobody had counted.

Every dependency has a limit, and the feature inherits all of them

An AI feature is rarely one call. It is a small chain: fetch data, maybe enrich it, call the model, maybe write somewhere. Each hop has its own limit, and the feature is only as generous as the tightest one. When people say “the AI feature,” they picture the model, so the model’s quota is the number they check. The enrichment API, the vector store, the internal service that returns customer records, each has a ceiling too, and any one of them can be the wall you hit first.

Rate limits also fail in a way that hides the cause. The call does not error in a way that says “you are over your hourly budget.” It returns a rejection that surfaces two steps later as a missing field or an empty summary, far from where the limit actually bit. So the feature looks flaky, and flaky is the word teams reach for when a system fails on a schedule they have not decoded yet.

What good teams do

List every external call the feature makes, not just the model. Each one is a dependency with its own limit.
Write down each dependency’s actual quota, per second and per hour, and compare it to your real peak load.
Handle the rejection where it happens. Catch the limit response, back off, and retry with jitter instead of hammering the window.
Cache what does not change. If the same shipment gets enriched twice, the second call spent budget you did not need to spend.
Alert on approaching a limit, not on crossing it. The useful moment is before the wall, while you can still slow down.

How we approach it at Density Labs

Our AI Opportunity Assessment is a fixed two-week engagement at $2,500 that scopes real production work before you build. For a feature like this, we trace the full call chain and put a number next to every external dependency, not only the model. Then we hold those numbers against the load the feature will actually see at peak. The wall is almost always a quiet data or enrichment API that nobody thought to budget, because attention went to the model.

The general finding is well known by now: close to 95 percent of AI pilots stall on integration rather than on the model. This is what that looks like up close. The model works. The thing feeding it ran out of allowance an hour ago, and the feature has been failing on a schedule ever since.

Budget every limit the feature depends on, not the one with the model’s name on it. The wall you did not count is the one you hit.