Cost per correct answer, not cost per call

Teams measure the wrong denominator. They track cost per API call, watch it fall as token prices drop, and conclude the feature is getting cheaper. But nobody buys a call. They buy a correct answer. And the cost of a correct answer can be several times the cost of a call, because getting to correct often takes more than one try, plus something to check the work. Track the call and the budget looks healthy. Track the correct answer and you see the real bill.

Where the two numbers split

A staff engineer at a networking-hardware company walked me through how his team got caught. Their per-call cost was tiny and kept shrinking, so they never worried about it. What they missed was their acceptance rate. Roughly a third of the model’s first answers were not usable, so the workflow retried, sometimes twice, before it got something it could ship. His per-call number was honest. It was also meaningless, because the thing that mattered was cost per usable answer, and that was closer to three times what the dashboard showed.

The split gets wider when you add verification. Output tokens cost around five times what input tokens cost on a Sonnet-class model, so a workflow that generates a long answer, then generates a critique of that answer, is paying the expensive rate twice. Every layer you add to reach correct multiplies the cost per correct, even as cost per call sits still.

A platform engineering lead at a SaaS company described the same trap from a quality angle. To trust the output, his team sampled the model several times and took the most frequent answer, a self-consistency approach that works well and costs real money. Five samples to produce one trustworthy answer is five calls per correct result. On the call-cost dashboard that looks like a five-fold regression. On the correct-answer dashboard it looks like the price of being right. Only one of those framings tells you whether to ship.

Why the call metric is so tempting

Cost per call wins because it is clean, it is available, and it moves in the direction you want. Providers quote it. Dashboards default to it. It falls over time, which feels like progress. Cost per correct answer requires you to define what correct means, measure how often you hit it, and count every attempt and check along the way. That is more work, and it produces a bigger, less flattering number. Under deadline pressure, the easy metric wins, and the feature ships on a cost model that was never real.

What cost per correct answer includes

To measure the number that actually matters, count all of this per accepted result:

First-pass acceptance. How often the very first answer is good enough to use, measured on real inputs.
Retry depth. The average number of attempts before the workflow produces something it keeps.
Verification spend. Any extra calls that check, critique, or vote on the answer before it ships.
Output weight. The generated tokens, priced at their real rate, since they dominate the bill on long answers.

How we approach it at Density Labs

Our AI Opportunity Assessment is a fixed two-week engagement, priced at $2,500. When cost is the concern, we define what a correct answer means for your use case, then measure what it truly costs to produce one, retries and verification included. That number, projected against production volume, is the one worth arguing about. It is usually far enough from cost per call to change the decision.

Cost per call is the price of trying. Cost per correct answer is the price of succeeding. Budget against the second one, because that is the only version your users ever pay you for.