Why your test suite goes green and the AI feature still regresses

A platform team at a customer-support company called me in after a bad week. They had shipped a change, watched CI go fully green, and rolled it to production with confidence. Two days later the complaints started. The AI assistant that drafted replies for agents had gotten worse. It was inventing order numbers, ignoring the customer’s actual question, and padding answers with filler. None of that showed up in a single failing test.

We sat down and read the suite together. Every test that touched the assistant mocked the model call. The mock returned a fixed, hand-written string that looked like a good answer. So the tests were checking that the code around the model behaved: that it parsed the response, stored it, and rendered it in the right place. Those things all still worked. The part that had changed was the model’s actual output, and no test in the suite ever ran the real model.

That is the trap. Deterministic tests are good at catching deterministic bugs. A wrapped model call is not deterministic, and the interesting failures live in the output, not in the plumbing.

Deterministic tests cannot see a behavioral regression

Here is what had happened underneath. Someone had tweaked the system prompt to make answers shorter. The change looked harmless. But shorter answers pushed the model to drop the step where it grounded its reply in the retrieved order data, and once that grounding was gone, it started guessing. The code path was identical. The prompt was different, the retrieved context was assembled slightly differently, and the model’s behavior shifted.

A mock cannot represent that shift. A mock is a photograph of one good answer, frozen in time. When you assert against it, you are asserting that your code can handle that one answer, not that the model still produces answers like it. So the suite stays green while the thing users actually experience gets worse. You are testing the frame around the picture and never looking at the picture.

The teams that avoid this add a second layer of tests that exercise the real model output and judge the behavior, not the exact string.

What good teams do

Keep a small golden set of real inputs with known-good expectations, and run them against the live model, not a mock.
Judge behavior with checks and rubrics: did the answer cite the retrieved order, did it stay on the customer’s question, did it avoid inventing IDs. Score properties, not exact wording.
Set thresholds instead of demanding a perfect match, so normal model variation does not turn every run red.
Run the behavioral evals on prompt changes and model-version bumps, the two edits most likely to move behavior with zero code diff.
Keep the mocked unit tests too. They still catch the plumbing bugs. Just stop trusting them to catch the model.

The support team already had the raw material for this. They had months of real tickets and the answers their best human agents had written. That was a golden set waiting to be used. We turned a few dozen of those into evals, wrote simple rubric checks for grounding and relevance, and wired them to run whenever the prompt or model changed. The next prompt tweak that dropped grounding failed loudly, before it reached a single customer.

How we approach it at Density Labs

Most teams do not need a bigger test suite. They need one that watches the model instead of the code around it. That is the kind of gap our AI Opportunity Assessment is built to find. It is a fixed two-week engagement, priced at $2,500, where we scope the real production work before you build anything. We read your existing tests, find where the model is mocked out of the picture, and design a small behavioral eval set from data you already have. You come out with a concrete plan for what to test, how to judge it, and where it plugs into CI.

Green CI should mean the feature still works. If your suite mocks the model, green only means the wiring survived. Test the behavior your users actually feel, and let the plumbing tests stay in their lane.