Testing a feature whose right answer is a range, not a value

A director of engineering at a recruiting platform told me their AI feature was stuck in testing limbo for weeks, and the reason was a question they could not answer. The feature ranked candidates for a role, and every time they tried to write a test, they hit the same wall. What is the correct ranking? There wasn’t one. A reasonable recruiter might order the top five candidates several different ways and be right each time, while ordering an obviously unqualified person at the top would be wrong. The team had been trying to assert one correct answer for a feature whose correctness was a range. Until they accepted that, they could not write a single meaningful test.

Not every feature has a single right answer

Some problems have exactly one correct output. Two plus two is four, and a test that asserts it is easy to write. A lot of engineering testing intuition comes from problems like that, where correct is a point and everything else is a bug.

Many AI features do not live in that world. Ranking, summarizing, drafting, recommending, classifying an ambiguous case, all of these have a band of acceptable answers rather than a single right one. Several outputs are fine. Some are excellent, some are merely adequate, and some are clearly wrong, and the line between adequate and wrong is the thing that actually matters. A team that keeps trying to pin down the one correct answer will keep failing to write tests, because they are asking a question the feature does not have an answer to. The output is not right or wrong. It falls inside the acceptable band or outside it, and testing means checking which side of the line it landed on.

The trap is that this feels like the feature is untestable. It is not. It is testable in a different shape. The work that feels missing is not test-writing, it is deciding, with the people who own the requirement, where the band is. What makes an answer acceptable, what makes it excellent, and what puts it clearly out of bounds. That decision is a product decision, and until someone makes it, the tests have nothing to check against.

Define the band, then test the edges

The lesson is that testing a range-valued feature starts with drawing the range, and that is a judgment call the team has to make on purpose. Once the band exists, the tests become tractable, and the most valuable ones check the edges.

What teams do to test a feature with a band of acceptable answers.

Define, with the requirement owner, what makes an output acceptable, what makes it good, and what makes it clearly wrong.
Test that outputs stay inside the acceptable band, rather than that they match one specific answer.
Build a set of clearly-wrong cases the feature must never produce, since the out-of-bounds region is where the real failures are.
Score quality inside the band with a rubric, so you can tell an adequate answer from an excellent one when it matters.
Accept variation within the band as correct, so the suite does not fail on differences that are genuinely fine.

The reason the feature felt untestable was that nobody had decided what correct meant. Deciding that is the work. The tests follow from it.

How we approach it at Density Labs

The stuck-in-testing AI feature is a pattern we see often, and the cause is almost always the same. The team is trying to write deterministic tests for a feature whose output is a range, and they are stuck because the range was never defined. They are waiting for a right answer that does not exist, when what they need is a decision about where acceptable ends and wrong begins.

Our AI Opportunity Assessment is a fixed two-week engagement, priced at $2,500, that scopes the real work of shipping an AI feature. Part of that is pinning down what an acceptable output actually is, with the people who own the requirement, so the feature has a testable standard instead of an argument that reopens every sprint.

A feature with a band of right answers is not untestable. It is untested until someone decides where the band ends.