Reproducing an AI decision six months after it happened

A compliance engineer at a regulated company told me about a request that exposed a gap nobody had planned for. An auditor pointed at a decision the AI feature had made months earlier and asked the team to demonstrate how it was reached. Reasonable ask. The team sat down to reproduce it and could not. The prompt had been revised several times since. The model version had been updated by the provider. The underlying data the decision drew on had changed. Running the same case today produced a different result, and there was no way to reconstruct the exact conditions that had produced the original one. The decision had happened, it had affected someone, and it was now impossible to reproduce or verify.

Reproducing an old decision is harder for AI features than for ordinary software, and teams rarely plan for it. With deterministic code, you can often reason your way back to what happened. With an AI feature, the behavior depends on the exact prompt, the exact model version, and the exact data at the moment of the decision, and all three drift over time. Unless you captured them as they were, the past is not recoverable, and you are left unable to answer how a specific decision was made.

Everything that shaped the decision keeps moving

Three things drift under an AI feature, continuously, and each one changes the outcome.

The prompt drifts because you improve it. Every revision changes behavior, which is the point, and it also means the prompt that produced a decision last quarter is not the prompt running now. The model drifts because the provider updates it, sometimes without a version change you controlled, so the same input can produce a different output than it did before. The data drifts because the world moves, records update, and context changes, so the information the decision drew on is not what it is today.

Run an old case through today’s system and you get today’s answer, shaped by today’s prompt, model, and data. That tells you nothing about how the original decision was reached. To reproduce it, you would need to restore the exact conditions of the moment, and if those conditions were not recorded, they are gone.

This is precisely what auditors, disputes, and investigations ask for. Not how the system behaves now, but how it behaved then, for this case. A feature that cannot answer that leaves you unable to account for decisions that already affected real people.

Version enough to reconstruct the past

The fix is to record, for each decision, the conditions that produced it, so you can reconstruct the past instead of re-running the present.

What to version and capture:

The prompt as it was. The exact prompt and logic in effect at the time, tied to the decision, so you know what instructions were running.
The model version. Which model produced the output, pinned and recorded, so a provider update does not erase the conditions of past decisions.
The inputs at the time. The data and context the decision drew on, captured as they were, since the live data will have moved.
The output produced. What the model returned and what final decision followed, so the record is complete end to end.

With those in hand, an old decision becomes reconstructable. You can show what prompt ran, on what model, against what data, and what came out. The teams that get this right decide early that decisions must be reproducible, because the request to reproduce one always arrives long after the moment has passed.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement, we check whether a past decision can be reconstructed. Whether the prompt, model version, and inputs were captured as they were, or whether re-running the case would only show current behavior. Reproducibility is a build-time decision, and it is one of the first things an auditor or a dispute will test. Capturing the conditions costs little at the time. Recovering them later is not possible.

An AI decision drifts out of reach the moment its prompt, model, or data changes. Version enough to reconstruct the past, because you will be asked to account for decisions long after they were made.