The regression test you can only write after the incident

A staff engineer at a travel company described the moment their AI feature testing philosophy changed. A customer had asked their booking assistant a question phrased in a way nobody on the team had ever considered, the assistant gave a confidently wrong answer, and it cost the company a refund and an apology. In the postmortem someone asked why there wasn’t a test for that case. The honest answer was that no one had imagined that case existed. You cannot write a test for a failure you have never seen. What the team could do, and had not been doing, was make sure they never met that particular failure twice.

You cannot pre-imagine every AI failure

For most software, you can enumerate the important cases up front. The inputs are structured, the paths are countable, and a thoughtful engineer can write tests that cover the ways the code can break before it ever breaks. Coverage is a goal you can actually approach, because the space of inputs is knowable.

AI features do not offer that. The input is often open-ended natural language, or messy real-world data, and the ways it can go wrong are effectively unbounded. Users phrase things in ways you never anticipated. The model latches onto a pattern you did not know it had learned. An input that looks ordinary produces an answer that is subtly and expensively wrong. You cannot write these tests in advance, because the whole problem is that you did not know they were possible. The failure space is not something you cover. It is something you discover, one incident at a time.

This changes what a test suite is for. For deterministic code, the suite mostly proves the known cases work. For an AI feature, the suite’s real value grows from the failures you have actually encountered, each one turned into a permanent test so it can never recur silently. The suite becomes a memory of everything the feature has gotten wrong, and that memory is only as good as your discipline in adding to it. A failure you fix without capturing is a failure you have invited back.

Make every failure a permanent test

The lesson is that with AI features, the eval set is not written once and finished. It grows from production, and the growth is the point. Every real failure should end its life as a test that guarantees the feature handles that case forever.

What teams do to turn incidents into coverage.

Capture every production failure as a test case, with the exact input, before the incident is closed.
Add the fixed case to a permanent evaluation set the feature must keep passing, so the failure cannot return unnoticed.
Treat “why wasn’t there a test” as a process question, answered by adding one now, not as blame for not predicting the unpredictable.
Group captured failures by kind, so a pattern of related cases points to a deeper fix, not just a patch per incident.
Run the growing eval set on every change, so the feature carries the memory of everything it has ever gotten wrong.

You will never anticipate every way an AI feature can fail. You can guarantee that each failure teaches the suite something, which turns painful incidents into permanent coverage.

How we think about it at Density Labs

The teams whose AI features get more reliable over time all do one unglamorous thing. They mine their failures. Every incident becomes a test, the eval set grows week over week, and the feature stops repeating its mistakes because the suite remembers them even when the people forget. The teams whose features keep surprising them are usually fixing failures without capturing them, so the same class of problem keeps coming back wearing a new phrasing.

When we help a team operate an AI feature, we build the loop from incident to test into the process, so a failure is not closed until it is captured. It is the cheapest reliability work available, and most teams skip it because nothing forces them to do it.

You cannot test for the failure you have not met. You can make sure you only meet each one once.