Retros for AI features and what a normal postmortem misses

A director of engineering at a media company told me about a postmortem that frustrated everyone in the room, because their whole method assumed things that were not true for this failure. Their postmortem process was solid and blameless. Build the timeline, find the change that caused it, identify the process gap that let the change through, add a safeguard. It worked well for normal incidents. This one was an AI feature that had degraded, and the method kept coming up empty. There was no single change that caused it, the feature had drifted as the input data slowly shifted. There was no clear moment, it had gotten worse over weeks. There was no process gap in the usual sense, nobody had done anything wrong. The postmortem was built to find a cause of a shape that this failure did not have.

Postmortems assume a deterministic story

The standard postmortem rests on a set of assumptions about how failures work. There was a change, something was different that caused the problem. There is a root cause, a specific thing that, if it had been different, would have prevented it. There is a moment, a point where it went wrong. There is often a decision or an action that, in hindsight, was the pivot. These assumptions come from deterministic systems, where failures usually do have a change, a cause, and a moment, and the postmortem method is very good at finding them.

AI failures frequently violate all of these. The failure can have no change on the team’s side, if the cause was the model updating or the input data drifting. It can have no single root cause, if it emerged from a gradual shift rather than a discrete event. It can have no moment, if it degraded slowly over weeks with no threshold crossed. And it can have no wrong decision to point to, if everyone acted correctly and the feature still failed. A postmortem method tuned to find a change, a cause, and a moment will keep coming up empty on a failure that has none of them, and the team will either force a false root cause to satisfy the method, or leave the retro feeling like they learned nothing. Both are bad. The forced root cause sends them fixing the wrong thing, and the empty retro wastes a real chance to learn from a failure that has genuine lessons, just not the shape the method expects.

The deeper issue is that AI failures often have lessons that are about the system’s ongoing behavior rather than a discrete mistake. The lesson is not what one person did wrong, it is that the feature needed monitoring it did not have, or that the team had no plan for the model changing, or that the input drift was foreseeable and unwatched. These are real, actionable lessons, and the standard postmortem, hunting for a change and a culprit, walks right past them.

Adapt the retro to how AI fails

The lesson is that retros for AI features have to be adapted to failures that may have no single change, no discrete cause, and no clear moment, or they will either invent a false root cause or find nothing. The method has to fit the shape of the failure.

What an AI feature’s retro asks that a standard one does not.

What drifted, not just what changed, since the cause may be a gradual shift in data or model rather than a discrete edit.
Why it took so long to notice, since silent degradation points to a monitoring gap more than a specific mistake.
What the feature needed that it did not have, like quality monitoring or a model-change plan, rather than who erred.
Whether the failure was foreseeable as a class, so the lesson is about the system’s design, not a single event.
How to detect this kind of failure faster next time, since for AI the detection gap is often the real finding.

The blameless spirit still applies, and it fits even better here, because AI failures often have no one to blame at all. The method just has to stop assuming a change, a cause, and a moment that the failure may not contain.

How we approach it at Density Labs

The postmortem that cannot find a root cause is a common and demoralizing experience with AI features, and it is not a failure of the team’s rigor. Their method is good. It was built for failures that have a change, a cause, and a moment, and AI failures often have none of those, so the method comes up empty on a failure that still has real lessons. The team either forces a false answer or leaves empty-handed, and both waste the incident.

When we help teams operate AI features, we adapt their retros to the way these failures actually happen, so the lessons about monitoring, drift, and model changes get surfaced instead of a hunt for a culprit who does not exist. It turns a frustrating retro into a productive one.

A postmortem looks for the change that caused the failure. Some AI failures had no change, and the lesson is somewhere your method was not looking.