Token and inference costs: why AI gets expensive at scale

There is a paradox at the center of AI cost. Per-token prices have collapsed, and bills have gone up. A GPT-4-class model dropped from about $20 per million tokens to about $0.40 in four years, roughly a fifty-fold cut. And plenty of teams paying those lower prices are spending more than ever. Cheaper inference did not save money. It removed the brake that was keeping usage down.

Cheap compute changes what you bother to optimize

A founder building an AI-native business intelligence platform captured the mindset that gets teams into trouble, and he was not wrong to hold it. “Now we have more computing power than we actually need,” he said. It sits idle, it is cheap, “so we can basically be ineffective on the computing power because who cares.” His mother, he pointed out, once had to schedule scarce university compute on punch cards. He can just spend it.

For a prototype, that instinct is correct. Shipping beats efficiency when compute is nearly free. But the same instinct scaled to production traffic is exactly how a bill balloons. Every inefficient call is trivial until you make it ten thousand times a day. The cost per call did not change. The number of calls did, and nobody was watching that number because each one felt like nothing.

The agentic multiplier nobody prices in

The sharpest version of this hides inside agentic workflows. A single agent that reasons in steps, calls tools, and retries can consume ten to thirty times the tokens of one simple request per user intent. Do the arithmetic on a modest five-step agent: ten thousand runs a day at two cents a run is $365,000 a year, for one workflow. That number is invisible in a demo, where the agent runs a handful of times for an audience. It is very visible in production, where it runs continuously.

The teams that get surprised did not misprice a token. They mispriced how many tokens one user action would actually trigger once the agent was doing real work.

Underneath the API is a physical bill

It helps to remember what an API call actually rents. A founder who runs an AI data-security company, and who watched the last wave of infrastructure up close, described the real substrate bluntly. The advantage sits with the few vendors that have “foundational models, energy to run them, a lot of GPUs, land, water to cool those huge data centers.” Inference feels like a line of code. It is GPUs drawing power in a cooled building somewhere, and at scale you are paying for that physical reality whether you priced it or not.

That framing matters because it changes the question. Cutting inference cost is not about finding a cheaper token. It is about doing less inference, or doing it more deliberately, because every call maps to real energy and real hardware downstream.

Where inference cost actually comes from

At scale, the bill is driven less by unit price and more by these:

Call volume. How many times per user action you actually hit a model.
Agentic depth. How many steps, tools, and retries each intent triggers.
Output length. Generated tokens cost more than input tokens, and verbose responses add up fast.
Idle inefficiency. The calls you make because compute felt free, not because they earned their place.

How we approach it at Density Labs

Our AI Readiness Assessment is a fixed two week engagement, priced at $2,500. When inference cost is the worry, we trace one user action all the way through to the number of model calls it triggers, then project that against real production volume. Often the fix is not a cheaper model. It is removing calls that felt free at prototype scale and stopped being free at production scale.

The price of a token kept falling. Your bill kept rising. The gap between those two facts is the number worth measuring.