The flaky demo was nondeterminism you had not named yet

A founder demoing his AI product to a room of prospects told me the story with a wince. He had rehearsed the flow a dozen times. Same input, clean output, every run. On stage, on that exact input, the model returned something broken and the demo stalled. Afterward the team decided it was bad luck, maybe a slow network, maybe he fumbled a step. None of that was it. The feature failed a fraction of the time and they had never once measured how often.

A dozen good runs is not a measurement

The rehearsals felt like proof because they kept passing. Here is the thing about a feature that fails, say, one run in ten: a dozen rehearsals can easily all land in the nine. You walk away certain it works. You have sampled the distribution a few times and seen only the good part of it, and you mistook that for the whole shape.

The founder had never run the same input fifty times and counted the failures. If he had, the stall on stage would not have been a shock. It would have been the expected one-in-something showing up at the worst possible moment, which is exactly when unmeasured risk tends to arrive. The demo did not get unlucky. It got a representative sample, live, with an audience.

That is the trap with a nondeterministic feature. Low-frequency failures hide from casual testing. A handful of manual passes is far too small a sample to see them, so the feature looks solid right up until it fails in front of someone who matters.

Characterize the variability before you rely on it

The fix is unglamorous and completely reliable. Before you demo or ship, take your real inputs and run each one many times. Count how often the output is acceptable. Now you have a number, and a number changes every decision that follows. A feature that passes ninety-something percent of the time is a different product from one that passes half the time, and you cannot tell which you have from a few good rehearsals.

Once you know the rate, the failure stops being a mystery and becomes a spec. You can decide whether it is good enough, add validation and retries to lift it, or hold the feature back until it clears your bar.

Here is the routine worth building.

Run every key input many times. Send the same input through repeatedly and record the outputs, so low-frequency failures actually show up in your sample.
Count the pass rate per input. Turn “it usually works” into a number for each case, so you know which inputs are shaky and which are solid.
Set the bar before the demo, not after. Decide the pass rate the feature must clear to be shown or shipped, and hold to it.
Add a validation gate and retry. Check each output and regenerate on a failure, so a shaky first pass becomes a reliable result the audience sees.
Demo on the same distribution you measured. Do not cherry-pick the input that always works in rehearsal, because the live run will not extend you that courtesy.

The point is to know your failure rate before it introduces itself. A measured feature can still fail, but it fails at a rate you chose to accept, not one you discovered on stage.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement, one of the first things we do is characterize the variability of the feature on real inputs. We run the cases many times and report the pass rate, because “it worked when we tried it” is not a measurement anyone should build a launch on. That number tells us whether the feature is ready, whether validation and retries will get it there, or whether it needs more work before a single customer sees it. Finding the failure rate in week one is far cheaper than finding it in front of a prospect.

A demo that passes every rehearsal and fails live did not get unlucky. You just never counted the failures.