Your prompt assumed a user who writes the way you do

A data scientist at an e-commerce retailer described a feature that worked beautifully in the office and disappointed in the wild. The team could not reproduce the failures. Every input they tried came back fine. Every input a real customer sent seemed to come back a little worse.

The difference was who was typing.

The prompt was written by people who write alike

The team that built the feature shared a way of writing. They were technical. They wrote in full sentences, in the company’s primary language, using the product’s own vocabulary. When they wrote the prompt and its examples, they used inputs that sounded like themselves, because those were the inputs they had. The examples read like an internal Slack message from an engineer, because that is effectively what they were.

Their customers did not write that way. Some wrote in a different language entirely. Some used shorthand and abbreviations. Some described products with the words a shopper uses, not the words the catalog uses. Some wrote three fragments and a question mark. Every one of these was a normal customer, and every one of them landed outside the narrow band of inputs the prompt was built to expect. Inside that band the feature was excellent. Outside it, the feature quietly got worse, and the band was much narrower than the team realized.

The prompt had not been tested against the users. It had been tested against the authors.

A prompt encodes assumptions about its inputs

Every prompt makes assumptions about the input it will receive. What language it is in. How long it is. How structured. How much context the user supplies. Which words they use for things. Those assumptions are rarely written down. They live in the examples you chose and the phrasing you used, absorbed from whoever happened to write them, and they shape the model’s behavior as strongly as any explicit rule.

When the real input distribution matches those assumptions, the feature looks great. When it does not, the feature degrades in ways the team cannot see, because the team keeps testing with inputs that fit the assumptions. The blind spot is built in. You wrote a prompt for a user who writes like you, and then you tested it as that user.

Closing the gap means going to the inputs you actually have.

Sample real user inputs. Pull actual traffic, not the inputs the team would produce, and read what your users really send.
Check the languages and registers. If customers write in another language or in shorthand, put those in your test set on purpose.
Use the users’ words. Test with the vocabulary your audience uses, not the internal product terms your team defaults to.
Include the short and messy inputs. Fragments, typos, missing context. These are the median, not the exception.
Measure across the distribution. Track where the feature is strong and where it falls off, so you know who it is quietly failing.

The aim is to replace an imagined user with the real one, before that real user meets the feature.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week engagement priced at $2,500, we insist on a real sample of user inputs, because a prompt tested only by its authors is tested against the wrong population. We look at the languages, the tone, and the structure your actual users bring, and we run the feature against that spread rather than against a demo. The failures usually cluster in the inputs the team never writes themselves.

Your prompt has an opinion about how your users write. Check it against how they actually do.

The users you built for are not always the users you have.