Snapshot tests break the day the model does its job better

A senior engineer at a marketing software company inherited an AI summarization feature with a wall of snapshot tests. Each test ran an input through the feature, captured the output, and saved it as the expected result. It looked thorough. The problem showed up when they tuned the prompt to produce better summaries. Every snapshot test failed, because every summary was now different, and different meant failed. The engineer had two choices, revert the improvement or regenerate all the snapshots, and regenerating them meant blindly accepting whatever the feature now produced as the new definition of correct. Neither option involved anyone actually checking whether the summaries were good.

A snapshot freezes the present as the standard

Snapshot testing is a reasonable idea for the right kind of code. You capture the current output, save it, and future runs compare against it, so any change gets flagged for a human to confirm. For a component that renders the same way every time, this catches unintended changes cheaply.

The technique assumes the current output is the correct output, and that assumption is exactly wrong for an AI feature. The whole point of an AI feature is that you keep improving it. You tune the prompt, you switch models, you add context, and the output changes on purpose, because you are trying to make it better. A snapshot test treats every one of those deliberate improvements as a regression, because it defined correct as “what it did last time,” and last time is precisely what you are trying to move past. The test does not encode a standard of quality. It encodes a memory of the output, and it fights you every time you improve it.

Regenerating the snapshots does not fix this. When you update a snapshot, you are declaring the new output correct, but nothing in the process made anyone judge whether it is. You could regenerate a snapshot that captured a worse summary, and the test would happily protect the regression from then on. The snapshot enforces consistency with the past. It has no opinion about quality, which is the only thing you actually care about.

Encode the standard, not the memory

The lesson is that a test for an AI feature should encode what makes an output good, not what the output happened to be last week. That is harder than capturing a snapshot, and it is the difference between a test that helps you improve and one that punishes you for it.

What teams use instead of frozen snapshots.

Define what makes an output acceptable, in properties and rubrics, and test against that standard rather than against a captured example.
Score outputs on a curated evaluation set, so a change that raises the score passes and one that lowers it fails.
Keep a small set of known-bad cases the feature must never regress on, and test that those stay handled.
When you do keep example outputs, treat them as illustrations for humans, not as the pass-or-fail bar.
Make improvement pass the suite by design, so tuning the feature is rewarded, not blocked.

The aim is a suite where making the feature better makes the tests happier. A snapshot suite does the opposite, and a suite that fights improvement will lose, usually by being ignored.

How we think about it at Density Labs

Snapshot tests on AI output are one of those things that look like diligence and function as a trap. The team feels covered because there are hundreds of tests. What they actually have is hundreds of frozen memories of old output, each one ready to fail the moment someone improves the feature, which trains everyone to regenerate rather than to check.

When we help a team put an AI feature on a solid testing footing, we replace the frozen snapshots with an evaluation that measures quality against a standard. The suite stops being a museum of past outputs and starts being a judge of whether the current output is good. That is the version teams keep, because it helps them move instead of holding them still.

A test should tell you the feature got worse. A snapshot only tells you it changed, and for an AI feature, change is the plan.