Replay: capturing enough to reproduce an AI failure after it happens

A principal engineer at a healthcare software company walked me through a bad afternoon. Their AI feature produced an answer that was wrong in a way that reached a customer and mattered. The customer flagged it. Leadership wanted it fixed and wanted assurance it would not happen again.

The team sat down to reproduce it. They ran the same feature on what they thought was the same input. They got a fine answer. They tried a few variations. Still fine. The bad output would not come back.

The trouble was that the request that failed no longer existed in any recoverable form. The exact input was partly gone. The context that had been retrieved and injected was gone. The parameters might have changed since. The model version might have moved. They were trying to recreate a moment with half the ingredients missing, and a fine result on a guess proved nothing.

The lesson: capture enough to replay any incident

An AI output is a function of several inputs, and to reproduce it you need all of them, not most. The exact input the user gave. The context or documents that were retrieved and stitched into the prompt. The parameters for that call. The model name and version. Miss one and the replay is a different call that happens to look similar.

So capture the full set, keyed to a request ID, at the moment of the call:

The user input exactly as received.
The retrieved context, or stable references to it with versions, so you can rebuild the same prompt.
The sampling parameters and any tool definitions in play.
The model name and version the provider reported.
The output that was actually returned.

With that, an incident becomes a replay. You reconstruct the exact call and run it. Now you can watch it fail, change one variable at a time, and know what fixed it. The healthcare team, after they added this capture, cut their next incident from an afternoon of guessing to a clean reproduction in minutes.

There is a wrinkle worth naming. Models are often nondeterministic, so even an exact replay may not fail identically every time. That is fine. A faithful replay still narrows the cause enormously, and it lets you test a fix against the real inputs instead of a made-up approximation. Some teams also pin parameters toward determinism specifically so incident replay is more reliable.

What good teams do

Capture the full input set for each call, not just the user’s text.
Store retrieved context or versioned references, so the same prompt can be rebuilt.
Key everything to a request ID that support and engineering both use.
Keep the capture long enough to cover the gap between a failure and its report.
Build a small replay tool so recreating an incident is one command, not an afternoon.

How we approach it at Density Labs

Irreproducible failures are among the most damaging AI production problems, because they block the fix and erode trust. Leadership asks whether it can happen again, and the honest answer is that nobody knows, since nobody can make it happen on purpose. That is the operations gap, not the model, and operations is where most enterprise AI efforts stall.

Our AI Opportunity Assessment is a fixed two-week engagement for 2,500 dollars that scopes the real production work before you build. For any AI feature that can affect a customer, part of that scope is confirming you capture enough to replay a failure, that the capture survives long enough to be useful, and that a fix can be tested against the real inputs rather than a guess.

If you cannot reproduce it, you cannot claim you fixed it. Capture enough to replay.