Why a cheaper model can cost more once you count the retries

There is a tempting move in every AI cost review. Swap the expensive model for a cheaper one and pocket the difference. On the per-token line it is an obvious win, sometimes a large one. The catch is that a cheaper model is often a less accurate model, and a less accurate model gets more answers wrong on the first try. Wrong answers trigger retries, verification, and human review, all of which cost money. Count those, and the cheaper model can end up costing more per correct answer than the one you replaced.

The saving that reverses

A staff engineer at a networking-hardware company ran exactly this experiment and kept the receipts. He moved a workflow to a model that cost a fraction of the previous one per token, and the first week’s bill dropped just as he expected. Then the downstream numbers moved. First-pass acceptance fell, so the workflow retried more often, and each intent now took more attempts to produce something usable. By the time he counted the extra calls, the cheaper model had erased most of its own saving. On the hardest inputs it had gone slightly negative. The per-token price was genuinely lower. The cost per correct answer was not.

The mechanism is worth being precise about. If a cheaper model needs to be sampled several times to reach a trustworthy answer, you are paying that lower per-token rate many times over for one result. Self-consistency, sampling the model repeatedly and taking the most frequent answer, is a real and useful technique, and it costs in direct proportion to how often the model disagrees with itself. A weaker model disagrees with itself more. So the saving on the token gets spent on the votes, and sometimes more than spent. Output tokens make it worse still, running around five times the cost of input tokens, so any extra generated attempt lands at the expensive end of the bill.

A CTO at a fintech had turned this into a standing rule for her team. No model swap was allowed to be judged on per-token price alone. The comparison had to be cost per accepted answer, measured on real inputs, with retries and verification included. Twice, she said, that rule stopped a swap that would have looked like a win on the invoice and lost money in practice. Once it confirmed a cheaper model really was better, because the accuracy held and the retries did not climb. The point was never that cheaper is bad. The point was that per-token price does not tell you which way the total goes.

Why the trap is so easy to fall into

The per-token comparison is seductive because it is clean and it is the number vendors put in front of you. It sits right there on the pricing page, it is easy to multiply, and it moves in the direction you want. Cost per correct answer takes work. You have to run both models on realistic inputs, measure acceptance and retries for each, and add up the full path to a usable result. That is more effort and it sometimes tells you the cheap option is not cheap, which is not the answer anyone hoping to cut the bill wants to hear.

Comparing models honestly

To compare two models without fooling yourself, hold these constant across both:

First-pass acceptance. How often each model’s first answer is good enough, measured on the same real inputs.
Attempts per success. The average number of paid calls each model needs to reach one usable answer.
Verification load. Any sampling or checking a model needs to be trusted, since a weaker model usually needs more.
Cost per accepted answer. The full path to one correct result, which is the only number that decides the swap.

How we approach it at Density Labs

Our AI Opportunity Assessment is a fixed two-week engagement, priced at $2,500. When a model swap is on the table, we run both options on your real inputs and compare cost per accepted answer, not cost per token. Sometimes the cheaper model wins cleanly. Sometimes its retries eat the saving and then some. Either way you get the number that actually decides it, before you commit a production workflow to a choice that only looked cheap.

The per-token price tells you what a call costs. It does not tell you what a correct answer costs. Compare the second number, because a model that is cheaper to call can be more expensive to be right with.