The confidence score users trusted more than they should have

A design lead at a company with an AI-assisted product told me about a signal that was doing the opposite of what they intended. Their feature showed a confidence score next to each answer, meant to help users judge how much to rely on it. The intention was good, giving people a sense of when to double-check. In practice, users read the number as a guarantee. A high confidence score got treated as a promise that the answer was correct, and people acted on those answers without checking, precisely because the number told them not to worry. The problem was that the score was not measuring what users assumed. It reflected the model’s own internal sense of certainty, which does not reliably track whether the answer is actually right. A confident wrong answer carried a high score, and users trusted it because of the number.

Trust signals shape how much people rely on an AI feature, and a signal that users misread builds trust in the wrong places. A confidence score that users interpret as accuracy, when it is really the model’s self-assessed certainty, tells people to trust exactly the confident mistakes they most need to catch. The number was added to help users calibrate. Because it did not mean what they thought it meant, it miscalibrated them instead, encouraging reliance on answers that had not earned it.

A signal is only as good as how it is read

The purpose of a trust signal is to help a user decide how much weight to give an answer. That only works if the signal means, to the user, what it means in fact. A confidence score creates a gap when the model’s internal certainty and the answer’s actual correctness are not the same thing, which they often are not. The model can be highly certain and wrong, and if the score reflects certainty, it will be high on those exactly-wrong answers.

Users, meanwhile, bring a natural interpretation. A confidence number reads as how likely this is correct. That is the intuitive meaning, and it is the meaning people act on, regardless of what the number technically represents. So they trust the high-confidence answers and doubt the low-confidence ones, when the thing they should be doing depends on whether the score actually predicts correctness, which the model’s raw certainty may not.

The result is misplaced trust with the appearance of calibration. Users feel they are being careful, because they are attending to the confidence signal. They are in fact being led, by a number that does not track accuracy, to rely most on the answers that are confidently wrong. A signal that miscalibrates is worse than no signal, because it replaces healthy uncertainty with false assurance.

Make the signal mean what users read

The fix is to make sure any trust signal you show actually corresponds to what users will take it to mean, and to design the presentation so it guides reliance correctly.

What that involves:

Show a signal that tracks correctness. If you display confidence, it should reflect how reliable the answer actually is, not just the model’s internal certainty, or it will mislead.
Match the presentation to the meaning. Present the signal in a way that supports the interpretation users will naturally bring, so what they read is what is true.
Encourage checking where it is warranted. For answers that should be verified, design the experience to prompt that, rather than a number that tells people to relax.
Do not show precision you do not have. A specific-looking score implies a precision that may not exist. If the signal is rough, present it as rough.

The teams that get this right treat trust signals as part of the safety design, because a signal that shapes reliance can cause harm when it shapes it wrongly. They show signals that mean what users read, and they are careful not to manufacture confidence the feature has not earned.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement, we look at the trust signals a feature shows and how users are likely to read them. Whether a confidence score tracks actual correctness or just the model’s certainty, and whether the presentation guides reliance toward the answers that deserve it. A signal that miscalibrates users is a common and quiet way an otherwise good feature erodes trust, because it looks like it is helping. We would rather catch that in a diagnosis than after users have relied on confident mistakes.

A trust signal users misread builds trust in the wrong answers. Make the number mean what people take it to mean, or it will calibrate them toward the confident mistakes.