When the safety check errors, which way does your AI fail

A backend engineer at a company with a moderated AI feature told me about an incident that came from a component working exactly as written, in the wrong direction. Their feature ran user requests through a safety check before answering, which was the right design. The check screened for the things the feature should not act on. One day the check itself had a problem, a dependency timed out, and the code did what code often does when a step fails. It moved on. The request proceeded to the feature without the safety check ever completing. So at the exact moment the safety layer was unavailable, the feature answered everything, unscreened, because failing open, letting the request through when the check errors, was the default nobody had chosen but everybody had inherited.

Every safety check faces a design question that teams rarely make on purpose. When the check itself fails, does the request proceed or stop. Failing open means it proceeds, so an error in the safety layer removes the safety. Failing closed means it stops, so an error in the safety layer blocks the request. The default in most code is to fail open, because that is what happens when you do not handle the error and the flow simply continues. That default is usually the wrong one for a safety check, because it disables the protection precisely when the protection is broken.

The default direction is the dangerous one

Think about when a safety check errors. It is not a calm moment. It is often during load, during a dependency problem, during exactly the kind of disruption where things are already going wrong. That is the worst possible time for your safety layer to quietly stop applying. Yet failing open means that is precisely what happens. The check breaks, the requests flow through unscreened, and the feature does the thing the check existed to prevent, at the moment it is least able to prevent it.

The reason this direction gets chosen by default is that it requires no decision. Code that does not explicitly handle a failed check tends to continue, because continuing is the path of least resistance. So the feature fails open not because anyone decided that unscreened requests are acceptable, but because nobody decided anything, and the absence of a decision is a decision to let requests through when the guard is down.

Failing closed has a cost, and that cost is real. When the check errors, requests are blocked, which means the feature is unavailable rather than unsafe. For most safety-critical checks that is the right trade, because a blocked request is a inconvenience and an unscreened one is a harm. But it is a trade, and it should be made deliberately, weighing what happens in each direction for your specific feature.

Choose the failure direction on purpose

The fix is to decide, for each safety check, which way it fails, based on what is worse when the check is unavailable.

What that involves:

Handle the check’s own failure explicitly. Do not let a failed safety check fall through to continuing. Decide what happens and write it, so the direction is chosen, not inherited.
Fail closed for the checks that matter. Where an unscreened request causes real harm, blocking on a check failure is usually the right trade, even though it costs availability.
Weigh the trade per check. Some checks guard against serious harm and should fail closed. Some are advisory and failing open is fine. Make the call based on the stakes.
Test the failure path. Deliberately break the safety check and confirm the feature fails the way you intended, because the failure direction is only real if it holds when the check actually errors.

The teams that get this right treat the failure direction as part of the safety design, not an accident of error handling. They know their checks will sometimes break, and they decide in advance whether a broken check means a blocked request or an unscreened one, rather than letting the default make that call at the worst moment.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement, we look at what happens when a safety check itself fails. Whether the feature fails open, letting requests through unscreened, or fails closed, blocking them. The default is usually to fail open, and it is usually the wrong default for a check that guards against harm. We would rather find that in a diagnosis than during the incident where the safety layer went down and everything flowed through it.

When your safety check errors, the feature fails one way or the other, and the default is the dangerous one. Choose the failure direction on purpose, so a broken guard blocks the request instead of waving it through.