Could and should are different filters for AI ideas

An innovation lead at an insurer told me about an idea that sailed through every review and should have been stopped at the door. The proposal was to use an AI model to adjust how aggressively the company pursued claims, based on a predicted likelihood that a given claimant would push back. Technically it was sound. The data was there, the model was accurate enough, the engineering was straightforward. On the feasibility scorecard, it was one of the strongest proposals that quarter.

It also would have quietly taught the company to lean hardest on the people least able to fight back. Nobody in the room framed it that way, because the room was answering a different question. They were asking whether the idea could be built. It could. The question of whether it should be built never got a gate of its own.

Feasibility is not permission

Here is the distinction the innovation lead started enforcing after that one got close. Whether you can build an AI feature is a technical question. Whether you should is a business and ethical one. They are different filters, and an idea can pass one cleanly while failing the other badly. The claims proposal passed the technical filter and failed the other, and because the process only had the technical filter, it nearly went through.

The reason this happens is that could is measurable and should is not. Feasibility has a scorecard. Accuracy, data availability, integration effort, cost. You can put numbers on it and rank proposals. Warrant has no scorecard. It asks who is affected, whether the outcome is one you would defend out loud, whether it survives contact with a customer who understands what you did. Those questions do not fit in a cell on a spreadsheet, so a process built around the spreadsheet skips them. The measurable gate crowds out the one that matters more.

And AI sharpens the stakes, because AI makes could cheap. Things that used to be too expensive to do at scale, like scoring every claimant’s likely resistance, become a weekend’s work. When could gets cheap, should has to do more of the filtering, and if you never built that gate, nothing filters at all.

A colleague who leads data at a hiring platform described the same near-miss. His team could have built a model that predicted which candidates would accept a lower offer, and used it to make lower offers to exactly those people. Feasible, accurate, and something the company would have been ashamed to explain to a candidate. The should gate caught what the could gate waved through.

Build the second gate on purpose

The fix is to make should a real, separate step in how you evaluate AI ideas, with its own owner and its own questions, not a vague hope that someone will raise a hand. Feasibility review tells you what you are able to do. A warrant review tells you what you are willing to do. Run both, and require an idea to clear each.

Questions the should gate asks that the could gate does not:

Who is affected by this, and would you defend the outcome to their face?
Does this create a result you would be uncomfortable explaining publicly?
Is the model being used to serve the customer, or to exploit something about them?
If a journalist described this feature plainly, would it still sound fine?

If an idea is feasible but you cannot pass it through those questions, it is not a project. It is a risk you were about to fund because it demoed well.

How we handle it at Density Labs

We run both gates in the AI Opportunity Assessment, our two-week fixed engagement at $2,500. We assess whether an idea can be built, and separately whether it should be, with the second question owned as deliberately as the first. Sometimes the most valuable thing we tell a client is that a technically excellent idea does not clear the should gate, and that is a conversation far cheaper to have during scoping than after a feature is live and a customer has noticed what it does.

Feasibility tells you an AI idea is possible. It does not tell you it is a good idea. Build the second gate, and make every proposal pass both.