Design review for AI: the meeting that has to happen before the sprint

A director of engineering at a logistics company realized, after a couple of AI features went sideways, that his team was missing a meeting. For normal features, jumping into the work was fine, the important decisions could be made as they went, and design review, when it happened, was light. For AI features, the decisions that determined success were nearly all made up front, before any code, and his team had no forum where those decisions got made deliberately. What data will this run on, and is it ready. How will we evaluate whether the output is good. What happens when the model is wrong. Who owns it when it breaks. These got decided by default, by whoever happened to make an implicit choice while coding, because there was no design review where they got decided on purpose.

AI features are decided before the code

For a lot of deterministic work, you can start coding and figure out the design as you go, because the consequential decisions are reversible and local, and a good engineer makes them well in the flow of the work. Design review exists, but for many features it is a formality, because the design is not where the feature lives or dies. The code is, and the code can be shaped iteratively.

AI features invert this. The decisions that determine whether an AI feature can ship are made up front and are expensive to reverse. Whether the data is ready and representative. How you will evaluate output quality, and against what standard. How the feature handles being wrong, since it will be. Where the human stays in the loop. Who owns it when it fails. How you will monitor it. These are design decisions, they are made before or early in the work, and getting them wrong is not a quick fix later, it is a rebuild. So an AI feature that is coded without these decisions being made deliberately is not being designed, it is being defaulted, with each implicit choice made by whoever hit it first, without the team weighing it. The feature’s fate is largely sealed by these upfront decisions, and if there is no meeting where they are made on purpose, they get made badly, by accident, and the sprint that follows is built on them.

This is why the light design review that works for deterministic features fails for AI ones. The consequential decisions are not in the code where iteration can catch them, they are in the design where a skipped review lets them slide by unmade. The team jumps into the work, the way that has always been fine, and the way that has always been fine skips the exact step that AI features cannot afford to skip.

Hold the design review before you build

The lesson is that AI features need a real design review before the sprint, because the decisions that determine whether they ship are made up front and are costly to reverse. The meeting that is optional for deterministic work is essential here.

What an AI feature’s design review covers before any code.

The data, whether it is ready, representative, and legally usable for this feature, since data readiness sinks more features than model quality.
The evaluation, how output quality will be measured and against what standard, decided before building, not after.
The failure handling, what happens when the model is wrong, since it will be, and how the feature fails safely.
The human in the loop, where a person verifies or intervenes, decided as a design input, not discovered in production.
The ownership and operability, who answers when it breaks and how it will be monitored, named before launch.

These are cheap decisions to make well in a meeting before the work and expensive to make badly by default during it. The design review is where an AI feature is actually decided.

How we approach it at Density Labs

The missing design review is behind a large share of the AI features we see stall or fail, and the cause is a habit that serves teams well elsewhere. For deterministic work, jumping in is fine, so teams jump into AI work too, and jumping in skips the upfront design decisions that AI features are decided by. The decisions get made by default instead of on purpose, and the feature’s fate is set before anyone realizes a decision was even being made.

This upfront design work is exactly what our AI Opportunity Assessment does. It is a fixed two-week engagement, priced at $2,500, that makes the consequential decisions, about data, evaluation, failure handling, human oversight, and ownership, deliberately, before the six-month build, so the feature is designed rather than defaulted.

For a normal feature you can design as you code. For an AI feature the design is made before the code, so the review has to happen before the sprint.