Free at prototype, expensive at production: the compute brake

Cheap compute changed a habit, and the habit is quietly expensive. When a model call costs almost nothing, you stop thinking about how many you make. That is the right instinct while you are prototyping, when shipping matters far more than efficiency. It is the wrong instinct in production, where the same casual call runs ten thousand times a day and each one draws real money. The brake that scarcity used to provide is gone, so usage climbs with nothing pushing back.

The instinct that helps then hurts

A founder building an AI-native business intelligence platform captured the mindset exactly. “Now we have more computing power than we actually need,” he told me. It sits idle, it is cheap, so his team could afford to be wasteful with it because, in his words, who cares. His mother once had to schedule scarce university compute on punch cards. He just spends it.

For a prototype, he is right, and I would not argue with him. When compute is nearly free, the fastest path to a working thing beats the efficient path, every time. But the exact instinct that makes prototyping fast makes production expensive. Every wasteful call is trivial until you make it at volume. The cost per call did not rise. The number of calls did, and nobody was watching that number, because each individual one felt like nothing worth counting.

A staff engineer at a networking-hardware company gave me the structural version of the same story. His team’s inference bill jumped one quarter with no new users and no new features. What had changed was a default. Someone had wired a helper model into a code path that ran on every request, because at prototype scale it was harmless and convenient. In production that convenience was a few million extra calls a month. The fix took an afternoon. Finding it took three weeks, because nothing looked broken. The feature worked. It just ran the model far more often than anyone had decided it should.

Why the brake disappears

Scarcity used to enforce discipline for free. When compute was expensive and limited, you rationed it without thinking, because you had to. Cheap inference removed that constraint, and it removed the discipline along with it. A GPT-4-class model fell from about $20 per million tokens to about $0.40 in four years, roughly a fifty-fold cut, and plenty of teams paying those lower prices spend more than ever. The price dropped. The bill rose. The missing piece is that nothing replaced the brake that scarcity used to be, so usage grew until the aggregate got someone’s attention.

Where the discipline has to return

To keep prototype habits from becoming production bills, put attention on these:

Per-request call count. How many model calls one user action triggers, watched as a number, not assumed to be one.
Default paths. The helper models and extra passes that got wired in when compute felt free and never got removed.
Volume projection. What a habit that is harmless at prototype scale costs once it runs at real traffic.
A cost owner. A named person who watches the aggregate bill, because the individual call is too small to prompt anyone on its own.

How we approach it at Density Labs

When we scope a feature that is moving from prototype to production, we trace one user action through to the number of model calls it fires, then project that against real volume. The savings almost never come from a cheaper model. They come from removing calls that earned their place at prototype scale and stopped earning it at production scale. That is a design decision, and it is much cheaper to make before launch than after the invoice.

Free compute is a gift to a prototype and a hazard to a product. The discipline that scarcity used to enforce for you has to come back as a decision, because nothing else is going to slow the calls down.