Concurrency, rate limits, and the ceiling nobody load-tested

Every AI feature has a limit on how many requests it can serve at the same moment. Model provider rate limits. Connection pools. Token-per-minute quotas. That limit is a real, physical ceiling, and most teams have never seen it, because a pilot never generates enough simultaneous traffic to touch it. Then a busy Tuesday does, and the feature that always worked starts timing out for everyone at once.

Sequential thinking, concurrent reality

The trap is that we design and test in sequence. One request, then the next, then the next. Real traffic is not sequential. It arrives in bursts, all at once, and the system has to hold many requests in flight together.

A semiconductor engineering leader framed the difference for me in terms his field knows well. You can characterize a component perfectly one signal at a time and still be wrong about how it behaves when every channel is active together. The interference, the heat, the shared resources under simultaneous load: those only show up when everything runs at once. Software has the same property. A feature that passes every one-at-a-time test can still have a concurrency ceiling nobody measured, because nobody sent it a crowd.

His rule was to test the crowd, not the individual. The individual always looks fine. The crowd is where the ceiling lives.

The provider has a limit too

There is a second ceiling that teams building on model APIs forget entirely. The limit is not just in your code. It sits in the provider’s terms.

A platform lead at a ride-hailing company hit this during a demand spike. Their AI feature called a model provider that enforced a tokens-per-minute quota. In normal traffic they never came close. During a surge, thousands of users triggered the feature in the same few minutes, the requests piled up against the quota, and the provider started rejecting calls. From the user’s side, the feature simply broke, and it broke worst exactly when the most people were using it. The team had load-tested their own servers. They had never load-tested the assumption that the provider would answer every call, because at pilot volume the provider always had.

That is the uncomfortable shape of a concurrency ceiling. It is invisible at low load, it is silent right up to the moment you cross it, and you tend to cross it at your busiest hour rather than your quietest.

Finding the ceiling on purpose

You want to meet your limit in a test, not in an incident:

Load-test the burst, not the average. Send many simultaneous requests, not a steady trickle. The peak is what breaks you.
Read every quota in the chain. Your provider’s rate limits, your database pool, your queue depth. The lowest ceiling wins.
Decide what a rejected request does. A graceful queue or a clear wait message beats a silent timeout when you hit the wall.
Watch concurrency, not just totals. Requests per day hides the peak. Requests in flight at once is the number that meets the ceiling.

How we approach it at Density Labs

When we look at whether a feature can scale, concurrency is one of the first things we pull apart. We map every rate limit and pool between the user and the model, find the lowest one, and ask what happens when real traffic pushes against it all at the same time. The point is to locate the ceiling deliberately, while it is a number on a page, instead of discovering it live when the room is full.

A ceiling you have measured is a capacity plan. A ceiling you have not is an outage waiting for your busiest day. The limit exists either way. The only choice is when you meet it.