Non-deterministic tests: making an AI feature’s pipeline pass reliably

A staff engineer at a fintech company showed me a CI dashboard that told a familiar story. The AI pipeline’s tests were red about a third of the time. Nobody had changed anything. The same commit passed one run and failed the next. The team had learned to shrug, hit rerun, and merge when it eventually went green. A few of them had started arguing to just delete the tests, because a test that fails randomly is worse than no test at all. They were not wrong about that. They were wrong about the cause.

The tests were flaky because they asserted exact string equality against model output. The model does not return the same string twice. So the test was checking something the model was never going to satisfy, and the only question was whether that particular run happened to land close enough. That is not a test. That is a coin flip with extra steps.

When a test is a coin flip, a team has two bad options and usually takes both over time. First they ignore red, because red stops meaning anything. Then they delete the test, because ignoring it made it worthless. Both leave the pipeline unguarded. The output can genuinely regress and nobody notices, because the signal was thrown out to stop the noise.

Test what should hold, not what the model happens to say

The way out is to stop asserting on the exact output and start asserting on the properties that should hold regardless of wording. A model answer can be phrased a hundred ways and still be correct. Exact match rejects ninety-nine of them. Property checks accept all the correct ones and reject the wrong ones, which is the actual job.

For the fintech pipeline, the checks that mattered were things like: the answer references the right account, it does not invent a number that was not in the input, it stays within the allowed topics, its structure parses. None of those care about phrasing. All of them fail when the model actually goes wrong. We replaced the string-equality assertions with those property checks and the flakiness dropped, because the tests were finally checking things the model could reliably satisfy when it was behaving.

We also cut the variance we did not need. Where the platform allowed it, we pinned temperature low and fixed the seed for the test runs, so the model’s own randomness stopped being a variable. That does not make the output deterministic, but it narrows the spread. And for judgments that needed nuance, we scored with a rubric instead of a hard match: a small set of graded criteria, with a threshold, so a slightly-worded-differently answer that was still correct passed, and a genuinely worse one did not.

What good teams do

Assert on invariants and properties: grounding, no invented values, allowed topics, valid structure. Not exact strings.
Pin what you can. Low temperature and a fixed seed shrink the model’s own variance in test runs.
Judge with rubrics and thresholds for anything subjective, so correct-but-differently-worded answers pass and worse ones fail.
Run each case a few times and require the property to hold across runs, so one lucky pass does not hide a real problem.
Keep the flake rate near zero on purpose. A test that cries wolf gets ignored, then deleted, then you are unguarded.

The team kept the tests instead of deleting them. More importantly, the tests started meaning something again. A red run now pointed at a real regression, because the random red was gone. Rerun-until-green stopped being the team’s merge strategy.

How we approach it at Density Labs

Flaky AI tests are usually a design problem, not a model problem. The model was always going to vary. The test was written as if it would not. Our AI Opportunity Assessment is a fixed two-week engagement, priced at $2,500, where we scope the real production work before you build. We look at where your pipeline is non-deterministic, find the assertions that are really coin flips, and redesign them around properties, thresholds, and rubric scoring so CI is green when the feature is good and red when it is not. You leave with tests your team will trust instead of rerun.

A model that varies is not a reason to give up on testing it. Test the things that should stay true no matter how the sentence is phrased, and red starts meaning something again.