Why an AI feature needs a dead-letter queue like any other integration

An ops team at a healthcare startup ran an AI feature that processed uploaded documents in the background. Most of the time it worked. Now and then a job failed, and the feature had one plan for failure: retry. A failed job went back on the queue and ran again. If the failure was temporary, a network blip, a busy provider, the retry fixed it, which is exactly what retries are for.

But some jobs failed for reasons a retry could never fix. A corrupted upload. An input the model rejected every time. A provider error tied to that specific request. Those jobs failed, went back on the queue, failed again, and went back on the queue. They spun forever, burning compute and, when the model was involved, money, on work that would never succeed. Worse were the ones that failed quietly and were dropped instead. Those documents simply never got processed, and nobody knew, because a dropped job leaves no trace. A patient’s record sat half-finished and no alert ever fired.

The feature could handle a temporary failure. It had nowhere to put a permanent one.

Permanent failures need a destination, not a retry

An AI job fails in two shapes. One is transient and clears if you try again. The other is terminal and will never clear no matter how many times you run it. A system that treats every failure as transient does one of two bad things with the terminal ones: it retries them forever, or it drops them. Neither leaves you with a job you can look at, understand, and fix.

A dead-letter queue is the destination for the second shape. After a bounded number of retries, a job that still fails moves off the main queue and into a place where it stops running and starts waiting for a human. That is the whole idea, and AI features earn it more than most, because their failures are varied and their retries can cost money. Any integration engineer would insist on a dead-letter queue for a payment processor or a message broker. An AI job is an integration with an external provider and unpredictable inputs, and it deserves the same treatment. The novelty of the model is not a reason to skip the boring reliability parts.

What good teams do

Separate transient from terminal failure. Retry the first with a bounded count and backoff; send the second somewhere else.
Give every AI job a dead-letter queue. After the retry budget is spent, the job moves off the main path.
Make the dead-letter queue visible. Someone should be able to see what is in it and why each job landed there.
Support replay. Once the cause is fixed, a person should be able to send the job back through cleanly.
Alert on the queue growing. A rising dead-letter count is a signal that something upstream changed.

How we approach it at Density Labs

Our AI Opportunity Assessment is a fixed two-week engagement at $2,500 that scopes real production work before you build. Failure handling for background AI jobs is one of the things we check, because it is easy to skip and hard to notice missing. We ask what happens to a job that fails permanently. The answer is often that nobody knows, which means the job either loops or disappears. Then we look for the dead-letter queue and usually find there is none.

The pattern is a plain version of the wider one: close to 95 percent of AI pilots stall on integration, not the model, and a feature with no place to put its permanent failures is stalling on integration whether or not anyone has named it yet.

Give your AI jobs the same dead-letter queue you would give any integration. A permanent failure should land somewhere a person can see it and replay it, not vanish or spin forever.