A pilot that cannot fail teaches you nothing
A pilot built to impress a stakeholder is designed to succeed, which is exactly why it tells you nothing about production. The useful pilot is the one that can come back with a no.
A pilot that cannot fail teaches you nothing
Most AI pilots are built to succeed. The inputs get hand-picked, the demo path gets rehearsed, and the audience sees the feature at its best. Then everyone declares the pilot a success and funds the build. A few months later the real system falls over on the cases the pilot was carefully steered around. The pilot was never a test. It was a performance, and a performance cannot fail.
A test you cannot fail is not a test
The founder of a developer-tools company described this from painful experience. His team was integrating payment providers that looked interchangeable at first. Then you get into the details, he said, and these services that look similar behave very differently, and suddenly you are drowning in edge cases. One provider would not let a subscription move from a yearly plan to a monthly one. You only find that when a real customer tries it.
A pilot that never lets a real customer try it will never surface that rule. It confirms what the team already believed and calls the confirmation evidence. The whole point of a pilot is to find the place the idea breaks while it is still cheap to change your mind. If you have designed it so nothing can break, you have spent the budget and learned nothing you did not already assume.
Design it so a “no” is possible
A semiconductor engineering leader gave me the sharper framing. In hardware, he said, you build the test to try to kill the part. You run it hot, you run it at the edge of spec, you feed it the input you are afraid of. If it survives that, you trust it. If it dies, you found the flaw for the price of a test coupon instead of a recall.
Software pilots rarely borrow that instinct, and they should. A pilot worth running has a real chance of returning a verdict you did not want. That means feeding it the messy records, the ambiguous requests, the inputs from the partner feed nobody trusts. AI generally needs around 85% data accuracy to be useful, and readiness is relative to the specific use case rather than some universal ready state. A pilot that only ever sees clean data will report readiness it has not actually earned.
A pilot designed to teach you something looks different from one designed to impress:
- It uses real inputs, pulled from logs and actual usage rather than curated to pass.
- It includes the cases you fear, the malformed record and the request at the edge of scope.
- It has a stated kill line, a result at which the honest answer is to stop.
- It measures against a baseline, so “better than what we do now” is a number, not a feeling.
The stakes are real. IDC found about 88% of proofs of concept never reach production, and many of those looked like successes right up until the moment they had to handle input nobody chose for them.
How we approach it at Density Labs
The AI Opportunity Assessment is built to be able to say no. Across a fixed two-week engagement at $2,500, we run the feature against the inputs a demo would avoid and we agree in advance on what a failing result looks like. Sometimes the outcome is that the feature is not worth building yet, and that is a finding worth far more than the fee, because it lands before the six-month build instead of after it.
A pilot that can only pass is a receipt for a decision you already made. A pilot that can fail is the only kind that changes your mind, and changing your mind cheaply is the entire reason to run one.