Batching vs real-time: the infra decision that sets your AI unit cost

A platform team at a marketing-software company ran all of their AI work through one path. Every request, whether a user was waiting for it or not, hit real-time inference the moment it arrived. That was the design from day one, and it never got revisited. When the feature grew, so did the bill, and the shape of the bill did not make sense to them until we sorted the traffic by whether anyone was actually waiting.

Most of it was not interactive. A large share of the requests were background jobs: enriching records overnight, scoring a backlog, generating summaries that nobody would read until the next morning. Those were being served by the same real-time infrastructure as the handful of requests where a user sat watching a spinner. The background work did not need an instant answer. It needed a correct answer by morning. But it was priced as if a person were waiting on every one.

That is the decision that quietly sets your unit cost. Not the model, not the prompt. Whether you force work to be real-time that could have been batched.

Separate what someone is waiting for from what can wait

Real-time inference is expensive because it optimizes for latency. You keep capacity warm and ready, you serve one request at a time so it returns fast, and you accept low utilization as the price of speed. That is the right trade when a human is waiting, because a slow answer is a bad experience. It is the wrong trade for work nobody is watching, because you are paying a latency premium to deliver a summary six hours before anyone opens it.

Batched inference makes the opposite trade. It groups many requests together, runs them through the hardware at high utilization, and accepts that any single one takes longer to come back. For background work that is exactly what you want. The same GPUs do far more work per dollar because they run full instead of near empty, and the added delay costs nothing because no one is waiting.

So the move is to split the traffic. We separated the truly interactive requests, the ones with a user on the other end, from the batchable ones, and gave each its own path. The interactive path stayed real-time, sized for its modest volume. The background work moved to a batched path that ran on a schedule at high utilization. Then we priced each path on its own, which the team had never been able to do before because everything was mixed together.

What good teams do

Sort requests by one question: is a human waiting for this answer right now?
Keep a real-time path for the interactive work, and size it for that volume alone.
Move background and bulk work to a batched path that runs at high utilization on a schedule.
Price each path separately, so you can see the unit cost of interactive versus batch and decide where to spend.
Set the batch window by the deadline that actually matters, not by habit. Overnight is fine if morning is the real deadline.

Once the paths were split, the batchable work cost a fraction of what it had, because it finally ran the hardware full instead of paying real-time rates to sit idle between requests. The interactive experience did not change at all. The users who were waiting still got fast answers, and the jobs nobody was waiting on stopped being charged as if someone were.

How we approach it at Density Labs

The batch-versus-real-time split is one of the decisions that most shapes the cost of an AI feature, and it is usually made once, by accident, and never revisited. Our AI Opportunity Assessment is a fixed two-week engagement, priced at $2,500, where we scope the real production work before you build. We sort your actual traffic into what needs an instant answer and what can wait, then design the separate paths and price each one, so you know exactly where your inference budget is going. You leave with a plan grounded in your real request mix.

Most AI work is not interactive, even when it runs on interactive infrastructure. Separate what someone is waiting for from what can wait, and the batchable half of your bill drops on its own.