Backpressure: what a slow model does to the queue behind it

An ops team at a healthcare startup shipped a document classifier that read intake forms and tagged them before a human reviewer saw them. In testing it was quick. Each form went in, the model answered, the form moved on. Then a partner clinic went live and started sending forms in the morning rush, and the service fell over in a way nobody had rehearsed.

The model call was slower than the rate forms arrived. Not by much. Forms came in a little faster than the model could clear them, and the gap did what gaps do. The queue in front of the model grew. Every waiting form held memory. Every new form pushed the oldest one further back. Latency climbed because a form now waited behind a hundred others before it even reached the model. Then the process ran out of memory and the scheduler restarted it, which dropped everything in flight and sent the retries back into the same queue that had just killed it.

Nobody wrote a bug. The arrival rate crossed the service rate, and an unbounded queue turned a small mismatch into a full outage.

The queue is the failure mode, not the symptom

When a model call is slower than the rate of arrivals, the queue behind it grows without bound. That is the whole story, and it is worth saying plainly because teams keep treating the slow call as the problem to fix. Making the call faster helps until traffic rises again. The queue is where the design has to hold.

An unbounded queue is a promise you cannot keep. It tells every caller “I will get to you,” and then it holds their request in memory while it fails to. The two things that break are the two things you were watching least: memory fills with waiting work, and latency stretches because the wait is the latency now. By the time a dashboard shows it, the queue is already deep.

The fix is to decide, in advance, what you will do when work arrives faster than you can finish it. You cannot make that decision during the incident. The queue has to have a ceiling, and hitting the ceiling has to mean something other than a crash.

What good teams do

Bound every queue. A queue with a maximum length turns “grow until we die” into “reject or divert once full,” which you can reason about.
Shed load at the edge. When the queue is full, reject fast with a clear signal so the caller can back off, rather than accepting work you cannot finish.
Apply backpressure upstream. Let the slow stage tell the fast stage to slow down, instead of letting the fast stage bury it.
Separate arrival rate from service rate in your own head. If arrivals can exceed service even briefly, the queue design has to survive that, not just the average.
Watch queue depth and wait time, not only the model’s own latency. Depth rising is the early warning the model’s latency will not give you.

How we approach it at Density Labs

Our AI Opportunity Assessment is a fixed two-week engagement, priced at $2,500, and it scopes real production work before you build. When an AI feature is in it, we look at the queue in front of the model as carefully as the model itself. We ask what the arrival rate looks like during a real peak, what happens when the model call slows down, and where the ceiling is. Most teams have never named that ceiling, so under load the queue grows until memory or patience runs out.

We map the path a request takes, mark the bounded queues and the shedding rules, and note where none exist. The finding is usually simple and hard to see from inside: the model is fine, the queue behind it has no floor. Naming the ceiling before launch is cheaper than discovering it during one.

A slow model is survivable. A queue with no ceiling is not. Decide what full means before traffic decides for you.