The webhook that never fired: async AI results and the systems waiting on them

A backend lead at a logistics company built a feature around a long-running AI job. The request kicked off the job, the job took a while, and when it finished the provider called a webhook to deliver the result. Clean design. The record would sit in a pending state until the webhook arrived and filled it in.

Most of the time the webhook arrived. Sometimes it did not. The provider fired it, the call failed for a moment, the provider gave up after a few tries, and the result was gone. On the company’s side, nothing announced this. The record stayed pending. There was no error, because from the feature’s point of view nothing had failed. It had asked for a job and was waiting for a callback that was never coming again. Downstream, other systems that read those records saw a job still in progress and waited too. The whole chain sat still, politely, on a result that had finished computing hours ago.

Support found it before the engineers did, because a customer asked why their thing had been “processing” since morning. It had not been processing. It had been done and then silently stranded.

Fire-and-forget assumes the callback always comes back

Asynchronous AI results are the right pattern for long jobs. You cannot hold a request open for minutes, so the job runs and reports back later. The hidden assumption is that “later” always happens. Webhooks fail. They get dropped in transit, rejected by a momentary outage on your side, or abandoned after the provider’s retry budget runs out. When the only path from “done” to “recorded” is that one callback, a single lost webhook leaves a record stuck in pending with no error and no owner.

The reason this hurts more than a normal failure is that it is silent and it spreads. A failed request tends to throw something a person can see. A missing webhook throws nothing, because waiting is a legitimate state. So the stuck record looks identical to a healthy in-progress one, and anything reading it inherits the wait. The system is not broken in a way that pages anyone. It is just quietly stopped, and quiet stops are the ones that reach the customer before they reach the dashboard.

What good teams do

Track job status explicitly. Record that a job was started, with an ID, so a pending record can be checked, not just trusted.
Reconcile on a schedule. Poll for jobs that have been pending too long and pull their result instead of only waiting for a push.
Set a timeout on waiting. A job pending past a sane limit should raise something, not stay pending forever.
Make webhook handling idempotent. A retried or duplicated callback should be safe, so you can afford to also poll.
Watch the age of pending records. A growing pile of old pending jobs is the symptom of lost callbacks.

How we approach it at Density Labs

Our AI Opportunity Assessment is a fixed two-week engagement at $2,500 that scopes real production work before you build. When a feature relies on asynchronous AI results, we ask one question first: what happens when the callback never arrives. If the only answer is “it does,” that is the finding. We look for status tracking, a reconciliation path, and a timeout on the wait, and we usually find the design assumes the webhook is reliable in a way webhooks are not.

The fix is not exotic. It is the same reconciliation discipline any team applies to an external system it cannot fully control. The model being on the other end does not change the shape of the problem, only the reason people forget to handle it.

Do not fire and forget an async AI result. Track the job, reconcile the ones that go quiet, and give every pending record a way to stop waiting.