The math of small error rates at large volume

A one percent error rate sounds like nothing. Say it in a review and everyone nods and moves on. Then the feature runs a hundred thousand times a day, and one percent is a thousand wrong answers before lunch. The rate did not change. What changed is how many times you multiplied it.

The number that feels small and the number that is large

Percentages hide volume. That is the whole trick. “Ninety-nine percent correct” reads like a grade you would be proud of, and it is, right up until you attach it to a denominator that keeps growing.

A leader who ran support operations at a consumer app described the moment it clicked for her team. In the pilot, the AI drafted replies for a few hundred tickets a day, and the handful it got wrong were caught by agents who were watching closely because the tool was new. The accuracy looked fine. Then the feature went to the full queue. Same model, same prompt, same one-in-a-hundred miss rate. The difference was that one in a hundred now meant dozens of customers a day getting a confidently wrong answer, and the agents had stopped watching because the tool was no longer new.

Nothing about the model degraded. The volume simply revealed what the pilot had been too small to show.

Errors are not evenly annoying

Here is the part the raw percentage never tells you. Some of those wrong answers cost you a shrug. Some cost you a customer, or a compliance finding you spend a month unwinding.

A payments engineering lead put it in plain terms. His fraud-scoring model was accurate on the vast majority of transactions, and the errors it did make were not distributed the way the average implied. The rare miss on a high-value transfer was worth a thousand routine misclassifications on small purchases. Averaging across all of them produced a number that looked calm and told him nothing about the risk that actually mattered. He stopped reporting a single accuracy figure and started reporting the error rate on the transactions where a mistake was expensive.

That reframing is the useful one. At volume, the question is not how often you are wrong. It is how often you are wrong in the cases you cannot afford to be wrong in.

What to check before the volume finds you

The gap between a clean pilot and a painful launch is usually one of these:

The denominator. Project the real daily request count, not the pilot’s. Multiply your error rate by that number and read the result out loud.
The cost of one error. Sort your outputs by what a wrong answer actually costs. The rare expensive miss deserves its own metric.
The catch rate. In the pilot, humans caught mistakes because they were paying attention. Decide who catches them once attention fades.
The distribution. A single average accuracy number hides the cases that end projects. Break it out by segment before you trust it.

How we approach it at Density Labs

Our AI Opportunity Assessment is a fixed two-week engagement at $2,500. When accuracy is the question, we do not stop at the model’s headline number. We take the error rate, multiply it by projected production volume, and separate the cheap mistakes from the expensive ones. Often the pilot’s accuracy was genuinely fine. The math only looked comfortable because the volume was small enough to hide what one percent becomes at scale.

A small error rate is a promise about proportion, not about count. The proportion holds as you grow. The count is what lands on your desk.