Who signs off when the model is wrong

An engineering director at a fintech told me about the meeting that changed how he scopes AI work. His team had shipped a feature that summarized customer documents. It worked well, until it summarized one badly and a customer acted on the summary. In the postmortem, the question that mattered was simple and nobody could answer it. Who signed off that this output was safe to show? The engineer said the model produced it. The product manager said engineering owned quality. The reviewer said they only checked the happy path. The mistake had an author. The accountability had none.

He put it plainly to me afterward. The model will be wrong sometimes, that part is a given. What is not a given is whether a human has already agreed to own being wrong. On his team, from then on, nothing shipped until one person had put their name on the standard for good enough and accepted that the misses were theirs to answer for.

Accountability does not diffuse just because the machine produced it

Non-deterministic systems tempt everyone to treat the output as ownerless. The model said it, not me. But a customer does not experience the model. They experience your company standing behind whatever the model showed them. Someone in your org is accountable for that whether or not you assigned it, and the only choice you have is whether they find out before launch or during the fallout.

The teams that handle this well decide the standard in advance. They do not aim for a model that is never wrong, because that model does not exist. They name the person who owns the acceptable-error line, the person who reviews the edge cases, and the person who answers the customer when a miss gets through. All three might be the same person on a small feature. What matters is that all three exist and have names.

Deciding the sign-off before you ship

Name the owner of the standard. One person defines what good enough means for this feature and accepts that the misses inside that line are theirs to defend.
Separate the author from the approver. The person who built the feature should not be the only one deciding it is safe to show a customer. Fresh judgment catches what pride hides.
Write down the acceptable failure. If the standard lives only in someone’s head, it changes silently under pressure. On paper, it holds.
Rehearse the wrong answer. Ask, before launch, what happens when a customer acts on a bad output. If that answer is improvised, you have not finished scoping.

How we approach it at Density Labs

In our AI Opportunity Assessment ($2,500), we make the sign-off explicit. Before the build, we help you name who owns the good-enough standard, who approves that the output is safe for a real user, and who answers when a miss gets through. It is uncomfortable to assign accountability for mistakes that have not happened yet. It is far more uncomfortable to assign it live, in a postmortem, with a customer already affected.

The model being wrong is not the risk you can remove. The risk you can remove is nobody having agreed to own it. Put a name on the miss before you ship the feature.