The changelog that skipped the prompt edit that broke everything

A staff engineer at a media company described an incident that took far longer to resolve than it should have, because the record they trusted was incomplete. When the AI feature started misbehaving, the team did the right thing and looked at the changelog to find the most recent change, on the theory that the last change is usually the cause. The changelog showed nothing relevant. No code deploy lined up with when the problem started. So they went looking elsewhere, spent hours ruling out data issues and infrastructure, and eventually found that someone had edited the prompt two days earlier through a config tool that did not write to the changelog. The change that caused the incident was real, recent, and completely absent from the record they used to find causes.

The changelog only records what deploys through code

A changelog, or a deploy history, is one of the first things a team reaches for during an incident, and for good reason. It answers the question that resolves most incidents fastest, what changed recently. The reliability of that answer depends on the changelog recording all the changes that matter, and for code-only systems it does, because the changes that matter are code changes and they all flow through the deploy pipeline that feeds the changelog.

AI features leak changes around the changelog. A prompt edited in a config store, a model version updated by the provider, a parameter tuned in a dashboard, none of these go through the code deploy pipeline, so none of them show up in the changelog. Yet all of them change the feature’s behavior and any of them can cause an incident. When one does, the team looks at the changelog, sees nothing relevant, and concludes the cause is not a recent change, which sends the entire investigation in the wrong direction. The changelog did not just omit the cause. It actively pointed the team away from it, by showing a clean recent history for a feature that had in fact just been changed. The most useful debugging shortcut becomes a source of misdirection, because it is complete for code and blind to everything else that drives the feature.

The cost is measured in the length of the investigation. The team rules out everything the changelog can see before they think to look at the layers it cannot, and those hours are spent because the record they trusted told them, falsely, that nothing had changed.

Put every behavior change in one record

The lesson is that the change history has to include everything that changes the feature’s behavior, not just code deploys, or it will mislead the team during exactly the incidents where it is most needed. One timeline, all the changes, regardless of which system made them.

What belongs in an AI feature’s change record.

Code deploys, as always, so the traditional causes remain visible.
Prompt edits, with author, timestamp, and the before-and-after, so a prompt change is as findable as a code change.
Model version changes, including provider-side updates, so a behavior shift from a new model is on the timeline.
Configuration and parameter changes, so a tuned setting that broke something is recorded, not lost in a dashboard.
One unified timeline, so during an incident “what changed recently” has a single, complete answer.

The goal is that the first place a team looks during an incident tells the whole truth about what changed, instead of the code-only slice of it. Right now, for many teams, the changelog is confidently incomplete, which is worse than obviously incomplete.

How we think about it at Density Labs

The changelog that hides the cause is a recurring and expensive pattern in AI incidents. The team does exactly the right thing, looks at the recent changes first, and the record misleads them because it only knows about code. The prompt edit, the model update, the config tweak, the things that actually caused the incident, are all invisible to the tool the team trusts most, so they spend hours looking everywhere except where the change was.

When we help a team operate an AI feature, we unify the change record so that every behavior-affecting change lands in one timeline. It turns the changelog back into what teams need it to be during an incident, a complete and trustworthy answer to what changed.

The last change is usually the cause. That only helps if your record actually contains the last change, and for AI features it often does not.