The evaluation you can run before you have a single user

The most common reason a team skips evaluation is that they think they need traffic to do it. They are waiting for real users to tell them whether the feature works. That wait is a mistake. A useful evaluation exists the day before launch, built from inputs you can write down yourself, and it catches the failures that would otherwise land on your first hundred customers.

You can manufacture the hard cases

The founder of an AI-native business intelligence platform described testing a query feature before anyone outside the company had touched it. He sat down with the two people who understood the domain best and had them write the questions a real analyst would ask. Not the clean ones. The vague ones, the ones with a typo in the metric name, the ones that assume context the model does not have. For each, they wrote what a right answer looked like.

That is a test set built from expertise instead of traffic. It does not need a single user because the knowledge of what good looks like already sits in the room. Most teams have that knowledge and never write it down, so it stays trapped in one senior person’s head until a customer trips over its absence.

The inputs that break it are the ones worth writing

A director of engineering at a messaging startup ran the pre-launch week differently. His team spent it trying to break the feature on purpose. Empty inputs. Inputs in a language the feature did not support. Inputs that told the model to ignore its own instructions. Inputs long enough to blow past the context window. He wanted the ugly cases found by his own team on a Tuesday, not by a stranger on launch day.

This matters more than the polish. MIT’s State of AI in Business 2025 found that roughly 95 percent of enterprise generative AI pilots deliver no measurable return. A share of those never faced a hard input until it was too late to matter. The feature demoed well, shipped, and met the first awkward question with a confident wrong answer.

What to build before launch

Write the golden cases with a domain expert, the real questions a user will ask, each with its expected outcome.
Add adversarial inputs on purpose, the empty, the hostile, the out-of-scope, the ones in the wrong language.
Score dimensions, not exact strings, so a non-deterministic output can still pass or fail on groundedness and relevance.
Keep every case, because the set you build before launch becomes the regression set you run after it.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week engagement, we help a team assemble this first set before the big build budget is committed. We pull the domain knowledge out of the people who hold it and turn it into cases with expected outcomes, then we push the feature with the adversarial inputs a demo would never surface. The team walks away able to answer a hard question: does this work, on the cases that matter, before a user has seen it.

Traffic is the slowest and most expensive teacher you can pick. The cheaper one is a room, two experts, and an afternoon spent writing down what a right answer means. Do that first.