The prompt that worked on three examples and broke on the fourth hundred

A backend engineer at a fintech company told me about a feature that looked finished for about a day.

The team had built a prompt that read a customer message and pulled out the fields their system needed. They tested it. It worked. They had three or four example messages on screen, pasted from a demo, and the model handled all of them cleanly. Format was right. Fields were right. Everyone agreed it was done, and they shipped it.

Then real customers started typing.

The inputs you test on are the inputs you wrote

The messages the team tested with were messages the team had written. They read like the person building the feature. Complete sentences. The fields in a tidy order. No typos, no missing information, no sarcasm. Every example was a message a developer would send, because a developer had sent it.

Real customers did not cooperate. One wrote in all lowercase with no punctuation. One left out the field the prompt assumed was always present. One buried the actual request under three paragraphs of backstory. One was annoyed and wrote like it. The prompt had never seen any of these shapes, so it guessed, and its guesses were wrong in quiet ways that nobody caught until support tickets piled up.

The prompt had not gotten worse. It was doing exactly what it did on day one. The inputs had changed, and the prompt was only ever tested against a version of the world that did not exist outside the demo.

A prompt is code that runs on inputs you have not seen

This is the part teams skip. A prompt is code, even though it reads like a paragraph you write once and forget. It runs, in production, against every input a real user can produce, and most of those inputs are nothing like the ones you had on your screen when you wrote it. Three happy-path examples tell you the prompt can succeed. They tell you nothing about how often it will.

The gap between “it worked on my examples” and “it works” is the whole job. You close it by looking at real inputs before you ship, not after the tickets arrive.

Here is what that looks like in practice.

Pull a real sample. Get a few hundred actual inputs from logs, past tickets, or a pilot, not the ones you would write yourself.
Run the prompt across the whole sample. Read the outputs. Sort by the ones that look wrong and find the pattern.
Test the ugly inputs on purpose. Missing fields, wrong language, a different tone, empty messages. These are normal, not edge cases.
Fix, then re-run the sample. A change that helps three examples can quietly break thirty others. You only know if you rerun.
Set a bar before you ship. Decide what accuracy on the sample counts as done, and hold to it.

None of this is exotic. It is the same discipline you would apply to any function that runs against untrusted input. The prompt just hides that it is one.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week engagement priced at $2,500, one of the first things we do is get our hands on a real sample of the inputs a feature will face. Not the demo messages. The actual traffic. We run the prompt across that sample and read where it fails, because that is the only honest measure of whether it is ready.

There is a rough industry bar worth keeping in mind: AI features generally need something like 85 percent accuracy or better on their real data before they hold up in production. You cannot know where you sit against that bar from three examples. You can only know it from the inputs your users actually send.

A prompt that passes on the examples you chose has told you nothing yet. Test it on the ones you did not.