Timeouts, retries, and the AI call that blocks a whole request

An engineering manager at a fintech company described an outage that still bothered him. One afternoon the whole API slowed to a crawl. Requests that had nothing to do with the new AI feature timed out. Health checks flapped. The team scrambled, expecting a database problem. The database was fine. The model provider had gotten slow for a few minutes, and a single call buried inside one request handler had taken the rest of the service down with it.

The provider recovered on its own. The damage was done by how that call was wired.

A blocking call with no timeout is a trap

Picture a model call made synchronously inside a request handler, with no timeout set. On a normal day it returns in a second or two and nobody notices. The trap springs when the provider stalls. The call does not fail. It waits. And while it waits it holds a worker thread, a connection, and whatever else that request had claimed.

Now the requests keep arriving. Each new one makes the same call, each one stalls, each one holds its worker. Your pool of workers is finite. It fills with requests that are all doing nothing but waiting on the same slow provider. Once the pool is full, every other request queues behind them, including ones that never touch the model at all. The feature’s problem has become the whole service’s problem. This is how a minor provider hiccup turns into a full outage.

Retries make it worse if you are not careful. A naive retry on a stalled call adds a second slow call on top of the first, and under load a wall of retries can push a struggling provider further down. You meant to add resilience. You added load at the worst possible moment.

Budget the timeout and free the thread

The lesson is that every model call needs a timeout, and no model call should hold a request thread hostage. Decide the longest you are willing to wait, set that as a timeout, and make the call cancellable so that when the budget is spent the worker is released instead of parked.

What this looks like in practice.

Every model call has an explicit timeout, chosen from a latency budget, never left to the default.
The call is made in a way that frees the worker while it waits, so a slow provider does not drain the pool.
Retries are bounded, with backoff and a cap, and never fired blindly at a provider that is already struggling.
A stalled call returns a fallback or a clean error, so the request finishes instead of hanging.
The AI path is isolated from the rest of the service, so its bad minute stays its own.

None of these are advanced. They are the basic hygiene of calling a slow external dependency, and a model provider is exactly that. The mistake is treating a model call like a fast local function when it is a network call to a system you do not control.

How we approach it at Density Labs

Most AI-related outages I hear about are not model failures. They are ordinary failures of calling a slow dependency without a timeout, letting one stalled call drain a shared pool. The model was innocent. The wiring around it was not.

Our AI Opportunity Assessment is a fixed two-week engagement, priced at $2,500, that scopes the real production work before you build. Part of that is looking at exactly how the model call sits inside your request path, what happens when the provider stalls, and how the call is bounded and isolated, so a slow minute upstream does not become an outage for you.

A model call is a network call. Give it a timeout and never let it hold the door.