Why ‘the model got worse’ is unprovable without output logging

An engineering lead at a large marketplace pinged me with a familiar worry. Their AI feature, which drafted product descriptions, felt worse than it did a month ago. Support had a few complaints. A couple of internal people said the same. The lead believed it. He wanted to fix it.

Then he tried to prove it, and could not.

There was no record of what the feature used to produce. The old outputs had been shown to users and discarded. He could see today’s output. He had nothing to compare it against. He could not say whether quality actually dropped, or whether a few loud complaints had shifted his perception. He could not tell what changed, because he could not see a before and an after.

So the team was stuck arguing about a feeling.

The lesson: store inputs and outputs so you can compare across time

A regression is a comparison. To claim the model got worse you need two things: what it did then, and what it does now, on comparable inputs. If you keep only the current output, you have half of every comparison and can prove nothing.

The fix is to store inputs and outputs over time. Keep the input that went in and the output that came back, with a timestamp and the model version attached. Then a suspicion becomes a query. You pull last month’s outputs and this month’s on similar inputs and look. Sometimes quality really dropped, and now you can see how and start on why. Sometimes it did not, and you just saved the team from chasing a ghost.

There is a second reason this matters, and it is one teams rarely plan for. Providers update models under you. A version you never chose to change can shift behavior. If you logged the version alongside the output, a quality complaint has an obvious first question: did the model version change between the good outputs and the bad ones? Without that record, you cannot even ask it.

What good teams do

Persist the input and the output for AI calls, not just transient in-memory values.
Attach a timestamp and the model version string to every stored output.
Keep enough history to compare across a provider change or a prompt change.
Sample rather than store everything if volume is huge, but sample consistently so comparisons stay fair.
Pair the stored outputs with any quality signal you have, like thumbs up or down, so you can find the bad ones fast.
Treat “the model got worse” as a hypothesis to test against the log, not a verdict to act on blind.

How we approach it at Density Labs

Perceived quality drops are one of the most common and most frustrating AI production problems, because without records they are unfalsifiable. The team cannot prove the regression, cannot rule it out, and cannot find the cause. They burn days on debate. It is another face of the pattern where AI efforts stall on operations rather than model quality, since the missing piece is not a better model but a record of what the model did.

Our AI Opportunity Assessment is a fixed two-week engagement for 2,500 dollars that scopes the real production work before you build. For a feature whose value is its output quality, part of that scope is confirming you keep enough history to prove a regression, trace it to a version or prompt change, and measure a fix against a real baseline.

You cannot fix a regression you cannot prove. Keep the outputs so the comparison exists.