What an error budget looks like for an AI feature

An AI feature that is right every time does not exist. You can push the failure rate down, but you will not reach zero, and pretending otherwise is how teams end up shocked by their own product a month after launch. Site reliability engineers settled a version of this argument long ago. They stopped chasing perfect uptime and started spending a budget for failure on purpose.

Borrow the idea from reliability engineering

An error budget is a number you pick before launch. It is the share of outputs you agree to let be wrong. Say the feature should be correct on 97 of every 100 cases. You now have a three percent budget, and that three percent is not waste. It is room to move. It pays for shipping something good enough instead of polishing something that never goes live.

An engineering leader who scaled large consumer systems described the discipline in plain terms. His teams treated the budget as spendable. When it sat full, they shipped faster and took bigger swings. When they burned through it, work stopped until quality climbed back. The budget was a throttle. It told them when to push and when to hold.

What counts as wrong has to be written down

A VP of engineering at a mid-market logistics company learned the hard part slowly. Her team launched a feature that summarized shipment exceptions, and for weeks nobody could agree whether a given summary was a miss. One reviewer flagged a dropped detail as a failure. Another read the same output and shrugged. The budget meant nothing because the definition of a burned unit kept sliding around.

They fixed it by writing the failure conditions down before counting anything. A summary that invents a shipment status is a hard failure. A summary that is thin but accurate is a soft one, tracked separately. Once the rule stopped moving, the budget turned into a real number they could watch from one week to the next.

Set the budget by what a wrong answer costs

Not every feature earns the same tolerance. A wrong movie suggestion costs a shrug. A wrong compliance flag costs a phone call from a regulator. The price of the bad output sets the number, and you cannot copy someone else’s budget into your own product.

Price the failure first, then set the budget. A high-cost wrong answer buys a tight budget and slower shipping.
Separate hard failures from soft ones. An invented fact and a merely weak answer are not the same debt, so do not count them the same way.
Track the burn over time, not a single score. A number that trends toward the ceiling week over week is the signal, long before you cross it.
Give the budget an owner. A number nobody watches is a number nobody defends when a deadline pushes on quality.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two-week engagement, one thing we help a team leave with is a first error budget grounded in what their wrong answers actually cost. We work out which failures are expensive, which are survivable, and what share of each the product can carry. It is a short conversation about numbers, and it is far cheaper to have before the build than after a bad output reaches a customer.

A feature with no error budget is not held to a higher standard. It is held to no standard at all, until the first failure sets one for you. Pick the number yourself.