The definition of done that never mentioned the model

A tech lead at a SaaS company told me about a feature that met their definition of done and was not remotely ready. Their definition was a good one, built over years. Code written, tests passing, reviewed, documented, deployed to staging. The AI feature checked every box. It also had no evaluation of its output quality, no monitoring for the ways it could degrade, no runbook for on-call, and no plan for what happened when the model changed. By the team’s own standard it was done, and by any honest standard it was half-built. The definition of done had been written for deterministic features, and it had no items for the things that make an AI feature actually ready.

Done encodes what readiness meant last time

A definition of done is a team’s compressed memory of what it takes to ship safely. Each item is there because, at some point, skipping it caused a problem. Tests, because untested code broke. Review, because unreviewed code broke. Documentation, because undocumented code cost someone later. The definition works because it captures the readiness lessons the team has already paid for.

The catch is that it captures lessons from the features the team has already shipped, and those were deterministic. So the definition assumes a feature is ready when the code is correct, reviewed, and deployed, because for deterministic features that is roughly true. An AI feature can satisfy all of that and still be missing the things that make it ready, because its readiness includes concerns the definition never had to cover. Is the output quality evaluated. Is quality monitored over time. Is there a runbook for a failure that gives wrong answers instead of erroring. Is there a plan for the model changing underneath you. Is there a kill switch. None of these are in a definition of done written before the team had ever operated a feature that varies and degrades, so an AI feature passes a bar that does not measure the things most likely to hurt it.

The danger is the false confidence. Meeting the definition of done is supposed to mean ready to ship, and the team trusts that, correctly, for most features. For an AI feature the signal is broken, it says ready when critical readiness work has not happened, and the team ships on a green light that was measuring the wrong things. The definition did not fail loudly. It just stopped covering the feature in front of it.

Add the items an AI feature needs

The lesson is that a definition of done has to grow to cover AI features, or it will keep certifying them as ready before they are. The additions are specific and they map to how AI features actually fail.

What an AI feature adds to the definition of done.

Output quality evaluated against a defined standard, not just tests passing, so done means the feature is good, not only that it runs.
Quality monitoring in place, so a feature that can silently degrade is watched on the axis it degrades along.
A runbook and on-call readiness, so the feature can be operated by someone who is not the person who built it.
A kill switch and a fallback, so the feature can be stopped fast if it does harm.
A plan for model and prompt changes, so a shift in the model or a prompt edit is a managed event, not a surprise.

These are not extras bolted onto a done feature. They are part of what done means for a feature that varies and can degrade, and until they are in the definition, done will keep meaning less than the team thinks.

How we think about it at Density Labs

The AI feature that meets the definition of done and is not ready is one of the most common gaps we find, and it is not a discipline problem. The team followed their definition faithfully. The definition was written for a kind of feature this one is not, so following it produced a feature that passed the bar and skipped the work that mattered most for AI.

When we help a team ship AI features, updating the definition of done is one of the highest-leverage things we do, because it changes the default. Once evaluation, monitoring, operability, and a model-change plan are in the definition, they stop being things a team remembers to do and start being things done requires.

A definition of done is only as good as the features it was written for. For an AI feature, most teams are shipping against a bar that was set before AI.