The same prompt, two answers, one confused user

A head of product at an education company forwarded me a complaint from a user that stuck with her. The user had asked the assistant a question, gotten an answer, and then asked the same question again a little later to double-check. The second answer was materially different from the first. Not a rewording. A different substance. The user’s message was short: which one is right, and why should I trust either?

That question is the whole problem. The feature had produced two answers to one question and had no idea it had done so. It had no memory that the question was a repeat, and no notion that a repeat should return a consistent result. Each call was independent, and independence is exactly what the user experienced as unreliability.

Nondeterminism is not an internal detail

The team knew, in the abstract, that a language model is nondeterministic. The same prompt can produce different outputs across calls. What they had treated as an internal, harmless property turned out to be something the user could see directly, because the user did the one thing that exposes it. They asked twice.

Most users never notice nondeterminism, because they ask a given question once. The ones who ask twice, to confirm something important or because they forgot they already asked, see two answers side by side. And two different answers to the same question read as a system that does not know what it is talking about. It does not matter that both answers might be defensible. The variation itself is the damage. Trust in an assistant rests on the sense that it has a settled view, and a feature that answers differently each time visibly does not.

For casual questions this is survivable. For the answers a user leans on, a policy, a definition, a recommendation they will act on, it is corrosive. The user is checking precisely because the answer matters, and the check is what breaks their confidence.

Decide where you need consistency

The lesson is that consistency is a design decision, not a default you get for free. You have to decide which questions demand a stable answer and build for it. The tools are not exotic:

Cache the settled answer. For questions that should return the same result, compute the answer once and serve the cached result on repeats. The second ask returns the first answer, because it literally is the first answer.
Lower the temperature where you want determinism. For outputs that should not vary, reduce the sampling randomness so repeated calls converge instead of wandering.
Pin the model and the prompt. Version them, so an answer does not shift because the underlying model or a prompt tweak changed between two asks.
Decide consistency per feature, not globally. Some features benefit from variety. Others must be stable. Name which is which and design each accordingly, rather than letting every answer be a fresh roll.

The principle is that a user who asks twice is a user you have to serve consistently, and consistency has to be engineered. The model will not give it to you on its own. You decide where a settled answer matters, and you build the caching or the determinism that makes the second answer match the first.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement that scopes real production work before you build, we ask a plain question of every AI feature: what happens when a user asks the same thing twice? For many features the honest answer is that nobody decided, so the result is a fresh, possibly different answer each time. We identify the questions that demand consistency and design it in, with caching, lower temperature, or pinned versions, so a repeat returns a stable result. It is far cheaper to decide this up front than to win back a user who caught the feature contradicting itself.

If a user can ask twice, they will. Make sure the second answer earns the same trust as the first.