How to build an evaluation set for your AI feature

An evaluation set is the small, curated collection of real inputs, plus the outcome you expect for each, that you run every change against. It is the difference between “it looked fine when I tried it” and “here is how it does on the 40 cases that matter.” Most teams either skip it or build it entirely from the happy path, which fails them at the same moment a demo does.

The value is in the cases that do not look alike

The founder of a developer-tools company described what happens when you build against surface impressions. His team was integrating payment providers, and at first they looked interchangeable. Then you get into the weeds, he said, and these services that look similar are actually very different, and now you have all these edge cases. One provider would not let a subscription move from a yearly plan to a monthly one, a rule you only discover when a real customer tries it.

That is the whole argument for an evaluation set. The inputs that look alike teach you nothing. The ones that diverge in some small, specific way are the ones that break in production, and they are the ones your test set has to contain. If you only collect inputs that behave the way you expect, you have built a set that confirms what you already believe.

He also made a quieter point that matters here. Something looked “not production ready” for years, when the real issue was that nobody had put in the effort to handle the awkward cases. Coverage is a choice, not a property you are handed.

Build it from real inputs, tested in isolation

A staff engineer at a networking-hardware company gave me the structural half. He is a strong believer in testing components in isolation behind a standard interface, so you can run one piece against known inputs and known expected outputs without the rest of the system interfering. That is exactly the shape an evaluation set should take: a specific input goes in, a specific expected result comes out, and nothing else is in the way to hide a regression.

For a non-deterministic AI feature, the expected result is rarely a single exact string, so you check dimensions instead. A useful set covers at least groundedness, whether the output is supported by the source, contextual relevancy, whether it answers the actual question, and guardrail adherence, whether it refuses or escalates when it should. Each case in your set carries those checks, and each change gets scored against all of them.

A practical way to assemble one

You do not need hundreds of cases to start. You need the right thirty or forty:

Pull real inputs from actual usage or logs, not inputs you invented to pass.
Add every failure you have already seen, so a fixed bug can never quietly return.
Include the awkward edge cases that diverge from the common path, the yearly-to-monthly kind.
Write the expected outcome per case as checkable dimensions, not a single golden string.

How we approach it at Density Labs

In the AI Readiness Assessment, our $2,500 front door, one of the artifacts we help teams leave with is a starter evaluation set built from their own inputs. We go looking for the cases that diverge, the ones a demo would never surface, and we write down what a correct outcome means for each so you can run any future change against them. It is a small file that saves you from relearning the same failure in front of a customer.

A demo is a test with one carefully chosen input. An evaluation set is a test with the inputs you were hoping nobody would try. Build the second one and keep it.