Versioning the part of your AI feature that is not in the repo

A tech lead at a fintech company told me they discovered a gap the hard way, when a customer disputed an answer their AI feature had given weeks earlier. The team wanted to reproduce it, to see exactly what the feature had done. They could pull the precise commit that was running in production at the time, down to the line. What they could not pull was the prompt that had been in effect, because prompts were edited in a config store with no version history, or the model version, because the provider had updated the model since, or the configuration, because it lived in a dashboard that only showed the current state. The code was perfectly versioned. Everything else that determined the behavior was gone. They could reproduce the code and not the feature.

Reproducibility requires versioning everything that matters

The reason engineers version code so carefully is reproducibility. If you know the commit, you can reconstruct exactly what the system was, which lets you debug an old incident, reproduce a reported bug, or roll back to a known state. Version control is the memory that makes a system knowable across time, and for code-only systems, versioning the code is enough, because the code is the system.

An AI feature is not fully described by its code. Its behavior is a product of the code, the prompt, the model version, and the configuration, and if any of those are unversioned, you cannot reconstruct what the feature actually did at a past moment. You can have the exact commit and still not know the behavior, because the prompt that shaped it has been overwritten, the model has been updated, the config has changed. The feature’s behavior at a point in time is only reproducible if every input to that behavior is versioned, and most teams version one of the four and leave the others floating. So they have the illusion of reproducibility, precise code history, over a feature whose behavior they genuinely cannot reproduce, because the parts that were not in the repo were never versioned and are simply gone.

This surfaces at the worst times. A customer disputes an old answer and you cannot show what the feature did. An incident traces to a past state you cannot reconstruct. A regression appears and you cannot compare against the version that worked, because that version was never captured. The unversioned layers are invisible until you need to look back, and then their absence is total, because you cannot version something after the fact.

Version the whole feature, not just the code

The lesson is that reproducing an AI feature’s behavior requires versioning everything that determines it, which means the prompt, the model version, and the configuration, alongside the code. Anything you leave unversioned is a part of the feature you can never look back at.

What versioning the whole feature includes.

The prompt, in version control with a full history, so any past prompt can be recovered exactly.
The model version in effect, recorded over time, so you know which model produced a given output even after the provider updates.
The configuration and parameters, versioned like code, so the settings that shaped the behavior are reconstructable.
A way to tie them together, so for any point in time you can say which code, prompt, model, and config were live at once.
The output records to match, so a past behavior can be examined against the exact versions that produced it.

The goal is that the feature’s behavior is as reproducible as its code, because you captured every input to that behavior, not just the one that happened to live in the repo.

How we approach it at Density Labs

The half-versioned feature is one of the most common and least noticed gaps in production AI. The team has excellent code versioning and assumes that means the feature is reproducible, when in fact the prompt, the model, and the config, the other three inputs to the behavior, are floating and unrecoverable. The gap is invisible until someone needs to look back, and by then the missing versions are gone for good.

Our AI Opportunity Assessment is a fixed two-week engagement, priced at $2,500, that scopes the real work of shipping an AI feature. Part of that is versioning the whole feature, prompt, model, and config included, so the behavior is reproducible when a dispute, an incident, or a regression makes you need to reconstruct exactly what the feature did.

You version the code so you can reproduce the system. For an AI feature, the code is a quarter of the system, and the other three quarters are usually not versioned at all.