The test suite that assumed the output would never change

An engineering lead at a customer-support software company told me about the morning their whole AI test suite went red. Nothing was broken. The model provider had shipped an update, the model started phrasing its answers slightly differently, and the tests, which asserted that the output matched an exact expected string, all failed at once. The new answers were as good or better. The suite had no way to know that, because it had been written to check for one specific string, and any change to that string read as a failure. The team spent a day rewriting expected values, and everyone quietly understood the tests had been measuring the wrong thing.

You cannot pin a non-deterministic output to a fixed string

Traditional tests work because most code is deterministic. The same input produces the same output, so you assert the output equals a known value, and any deviation means something broke. That contract is the foundation of the entire testing discipline most engineers grew up with.

AI features break the contract. A language model does not promise the same output for the same input, and even when you hold the input steady, the model behind it changes underneath you when the provider updates it. Asserting that the output equals a fixed string is asserting that a non-deterministic system will behave deterministically, which it will not. So the test fails not when the feature is wrong, but when the output changes at all, including when it improves. You have built a tripwire that fires on any movement, and most movement is not a problem.

This produces the worst kind of test, the one that cries wolf. When the suite goes red for reasons that are not failures, people stop trusting it. They rewrite the expected values to make it green again, which trains the team to treat red as noise. Then a real regression comes through, the suite goes red the same way it always does, and someone updates the expected value without looking, because that is what you do when this test fails. The brittle test did not just fail to catch the problem. It taught everyone to ignore the signal that would have.

Test the properties, not the exact answer

The lesson is that testing a non-deterministic feature means asserting on the qualities the output must have, not on one exact output. The right answer is a set of things that must be true, not a single string, and the tests should check those things.

What teams do to test AI output without pinning it to a string.

Assert on properties the answer must have, such as containing the right value, staying in a valid format, or not exceeding a length, instead of an exact match.
Check structure and constraints, so the test passes for any correct answer and fails for wrong ones, regardless of phrasing.
Use a graded evaluation for quality, scoring outputs against a rubric, rather than a pass-or-fail string comparison.
Separate “did the output change” from “did the output get worse,” so an improvement does not read as a regression.
Pin the parts that are supposed to be exact, like a returned identifier, and leave the parts that are supposed to vary free to vary.

The goal is a suite that goes red when the feature is actually wrong and stays green when the output legitimately changes, which is the only kind of suite a team will keep trusting.

How we approach it at Density Labs

The brittle AI test suite is one of the most common things we find when a team’s AI feature keeps surprising them in production. The tests were written in the deterministic style everyone knows, they break on every model update, the team learns to ignore them, and then the feature has no real safety net at all. The suite exists, it is green most of the time, and it is testing nothing that matters.

Our AI Opportunity Assessment is a fixed two-week engagement, priced at $2,500, that scopes the real work of shipping an AI feature. Part of that is designing tests that survive the non-determinism, so the suite tells you when the feature broke instead of when the model paraphrased itself.

A test that fails when the output improves is not protecting you. It is training you to ignore it.