The 95% is an engineering statistic, not a model one

MIT’s State of AI in Business 2025 study found that roughly 95% of enterprise generative AI pilots return nothing measurable. Read the headline and you assume the models came up short. They usually did not. In most of these pilots the model performed fine in the room where someone demoed it. What failed lived downstream, in the plumbing nobody scoped.

Where the failure actually lives

A VP of engineering at a mid-market logistics company put it plainly. Their pilot summarized shipping exceptions and it looked sharp on a stage. Then it met the real data. Half the records came from a partner feed that arrived late and malformed. The model never saw clean input twice in a row. Nobody had budgeted for the reconciliation layer that would have fixed that, because the demo had been fed a tidy sample by hand.

That is the pattern behind the statistic. The model is one component in a system that has to fetch data, handle a bad response, respect access rules, and stay up when traffic climbs. IDC found that about 88% of AI proofs of concept never reach production. A model that scores well on a benchmark tells you almost nothing about whether that surrounding system exists. Most of the time it does not, and building it is the expensive part.

The model is the cheapest part

A staff engineer at a networking-hardware company gave me the other half. He builds everything behind a clean interface so each piece can be tested against known inputs on its own. When he looked at a failing AI feature, he did not touch the model first. He looked at the seams. The retry logic swallowed errors. The context being passed in was stale. The place where the feature called out to another service had no timeout. None of that is a modeling problem. All of it decides whether the feature works on a Tuesday afternoon under real load.

He said something I keep repeating. A model you can swap in an afternoon. The system around it takes months, and that system is what you are actually shipping.

Teams that beat the odds tend to treat the model as settled early and spend their attention elsewhere:

They scope the data path first, including where the ugly inputs come from and who cleans them.
They design the failure behavior, what the feature does when a call times out or returns nonsense.
They test components in isolation, so a regression shows up in one seam instead of the whole flow.
They name the production owner up front, the person who answers when the feature misbehaves live.

Vendor-led builds reach production about 67% of the time, roughly double the rate of internal-only efforts, and the difference is rarely model quality. It is that an outside team has usually been burned by the plumbing before and scopes for it.

How we approach it at Density Labs

Our AI Opportunity Assessment is a fixed two-week engagement at $2,500, and it deliberately spends very little time on the model. We assume a capable model exists. What we map is the path from real input to shipped output: the data feeds, the failure handling, the integration seams, the owner. That map is where the 95% is decided, and it is far cheaper to draw before the build than to discover mid-build.

Blaming the model is comfortable because the model is the part you can name. The failure rate stays high because the parts nobody names are the ones that break.