Explaining an automated decision you cannot explain

A legal counsel at a company operating under strong data protection rules told me about a request that revealed a design gap. A person subject to an automated decision had a right, under the applicable law, to meaningful information about how that decision affecting them was made. The person exercised it. The team gathered what they had. They could describe the system in general terms, how it worked overall, what kind of factors it considered. What they could not do was say anything meaningful about this person’s specific decision. Why theirs went the way it did. What about their case drove the outcome. The general description was not what the law asked for, and it was not what the person wanted. Both required case-level explanation, and the feature had been built to produce decisions, not accounts of them.

Where the law grants rights around automated decisions, it generally asks for more than a brochure about the system. It asks for meaningful information about the specific decision that affected the specific person. That is a higher bar, and it is a bar you can only clear if the feature was built to produce case-level explanations. A general description of how the model works does not tell a person why they, in particular, were denied. Meeting the right means being able to speak to their case, not just to the system in the abstract.

General and specific are different obligations

There is a real difference between explaining a system and explaining a decision. Explaining the system is describing how it works in general. That is useful, and it is often part of what you owe, but it does not answer the question a person actually has when a decision goes against them. They want to know about their case. What factors mattered for them. What led to their outcome. That is case-level information, and it can only come from having captured the basis of their particular decision.

Many features cannot produce it, because they were never built to. The decision was made and recorded as a result, without the reasoning that would let anyone explain it afterward. So when the request comes, the team can offer only the general description, which is the part they can always produce and the part that does not satisfy the obligation. The specific explanation is missing because the specific basis was never kept.

This matters beyond compliance, though compliance is often what forces the issue. A person who receives a life-affecting decision and is given only a generic description of the system is right to feel that they have not really been told anything. The obligation to explain is, at bottom, an obligation to treat people as owed an account of decisions made about them, and a general description does not discharge it.

Build for case-level explanation

The fix is to design the feature so that each decision comes with enough captured basis to explain that specific decision to the specific person.

What that requires:

Capture the basis per decision. Record the factors that drove each individual outcome, in a form that can become a meaningful explanation of that case.
Keep the key factors interpretable. For decisions that affect people significantly, favor approaches where the main reasons can be named honestly, so the case-level explanation is real.
Distinguish the two obligations. Be able to describe the system in general and to speak to the individual decision, because the law and the person usually want both.
Know the rights that apply. Understand what your jurisdiction requires for automated decisions about people, because the required depth of explanation shapes what you must capture.

The teams that get this right treat the explanation as part of the decision, generated with it and kept with it. When the request arrives, they can speak to the person’s actual case, which is what the right was for and what the person deserves.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week, $2,500 engagement, we look at whether the feature can explain individual decisions or only describe the system. Where rights around automated decisions apply, we check what case-level basis is captured and whether it would meet the obligation. Designing for meaningful explanation is a choice made early, because the basis for a decision cannot be recovered once the decision has passed and the request is already in hand.

A right to explanation is a right to know about your case, not to read a description of the system. Build so every decision carries the basis to explain that specific one.