Testing the unhappy path an AI feature invents on its own

An engineering manager at a property management company told me their AI feature had thorough error handling and still embarrassed them in production. The team had tested the unhappy paths they knew, the API timing out, the input being malformed, the downstream service returning an error. All of that was handled cleanly. What they had not tested was the failure the model itself introduced. The feature answered a tenant’s question by confidently inventing a policy that did not exist. Nothing errored. No exception was thrown. The code worked perfectly and produced a wrong, invented answer, and their entire notion of an unhappy path was built around the code failing, not the model making something up.

AI features have a second category of failure

For ordinary software, the unhappy paths come from things going wrong mechanically. Something times out, something returns an error, something is null, something is out of range. Good error handling anticipates these and responds gracefully, and a thorough team can enumerate them and test each one.

AI features add a whole second category of failure that has nothing to do with the code working. The code can execute flawlessly and the model can still produce a bad output. It can invent a fact with total confidence. It can answer a different question than the one asked. It can drift off topic, contradict a source it was given, or produce something plausible and wrong. None of these throw an exception. None of them look like a failure to the code, because the code did its job, it called the model and returned the model’s answer. The failure is in the content, and content failures are invisible to error handling that watches for mechanical problems. A team that tests only the mechanical unhappy paths has tested half the ways the feature can hurt them, and skipped the half that is unique to AI.

This is a real blind spot, because it is so easy to feel covered. The team did test the unhappy path. They tested every unhappy path they had a mental model for, and their mental model came from years of non-AI systems where failure meant something broke. The model failing while everything works is not in that model, so it is not in the tests, and it walks straight into production.

Test the model’s failure modes explicitly

The lesson is that testing an AI feature means testing the ways the model can fail, on top of the ways the code can fail. Those are different exercises, and the model failures are the ones your existing instincts will miss.

What teams do to test the model’s unhappy paths.

Test for invented facts by feeding inputs where the correct answer is “I do not know,” and checking the feature does not fabricate one.
Test for off-topic and out-of-scope inputs, and check the feature declines or redirects instead of confidently answering.
Test against the source, for grounded features, and check the output does not contradict or exceed what the source supports.
Test adversarial and manipulative inputs, since users will try to push the feature into saying things it should not.
Treat a confident wrong answer as a failure worth a test, even though nothing errored, because that is the failure customers actually hit.

The code’s error handling is still necessary. It just does not touch the category of failure that makes AI features risky, and that category needs its own tests.

How we approach it at Density Labs

The AI feature with great error handling and an embarrassing production incident is a pattern we see constantly. The team was diligent about the failures they knew, and those failures all came from a pre-AI world where things break by erroring. The failure that got them was the model working exactly as coded and inventing something, which their error handling was never designed to notice.

Our AI Opportunity Assessment is a fixed two-week engagement, priced at $2,500, that scopes the real work of shipping an AI feature. Part of that is mapping the model’s failure modes, the confident wrong answer, the invented fact, the off-topic reply, and building tests for them, so the feature is checked against the ways AI actually fails and not just the ways code does.

Your error handling catches the code breaking. It will not catch the model making things up. That needs a test of its own.