Circuit breakers for a model provider that goes down

An engineering lead at a large retailer walked me through an outage that was worse than it should have been. Their model provider had a rough hour. Elevated errors, slow responses, timeouts. That alone was survivable. The AI feature could have degraded and the rest of the product could have stayed up.

Instead the whole checkout flow got slow, because the AI step sat in the middle of it. Every request hit the struggling provider, waited for a long timeout, failed, and retried. The retries piled onto a provider that was already overloaded. Threads and connections backed up. The failure spread outward from one feature into paths that had nothing to do with AI.

One dependency was sick. The naive retry logic turned that into a product-wide slowdown.

The lesson: trip fast, fall back, recover automatically

A circuit breaker is an old pattern and it applies cleanly here. When calls to a dependency start failing past a threshold, the breaker trips. While it is open, you stop calling the failing dependency and go straight to your fallback, instead of making every request wait for a timeout and then retry into the fire. After a cooldown, the breaker lets a trial request through. If the provider has recovered, it closes and normal traffic resumes. If not, it stays open a while longer.

This does three things at once. It protects your product, because a sick provider stops consuming your threads on doomed calls. It protects the provider, because you stop hammering it while it is trying to recover. And it recovers on its own, without a human watching a dashboard at 2 a.m.

Retries still have a place, but they need discipline. Retry a small, bounded number of times, with backoff, and only for errors worth retrying. Then let the breaker take over. Unbounded retries into an outage do not add resilience. They add load to the thing that is already failing.

What good teams do

Put a circuit breaker around every external model call.
Set a failure threshold that trips the breaker before the whole feature is waiting on timeouts.
Route to a defined fallback while the breaker is open, rather than queuing more doomed calls.
Bound retries with a small cap and backoff, and never retry into an open breaker.
Set aggressive timeouts, since a slow provider ties up resources as surely as a failing one.
Let the breaker probe and recover automatically once the provider is healthy again.

The retailer added a breaker after that outage. The next time the provider wobbled, the breaker tripped, the AI step fell back to its non-AI path, and checkout stayed fast. Most customers never knew the model was unavailable.

How we approach it at Density Labs

Provider dependency is one of the risks teams underweight until it bites them. The model works, the feature ships, and nobody tests what happens when the provider has a bad hour. That is an operations and resilience gap, the same territory where most enterprise AI efforts stall rather than on model quality.

Our AI Opportunity Assessment is a fixed two-week engagement for 2,500 dollars that scopes the real production work before you build. For any AI feature that calls an external provider, part of that scope is confirming there is a circuit breaker, a bounded retry policy, tight timeouts, and a fallback the breaker can lean on, so a provider outage stays contained to the feature.

Assume your provider will have a bad hour. Trip fast, fall back, and recover without a human.