What happens to your AI feature at 100x the requests

A VP of engineering at a healthcare startup walked me through a feature that had run cleanly in a closed beta for a month. A few dozen users, a steady trickle of calls, no problems. Then marketing scheduled the general launch. He wanted a second set of eyes before the traffic arrived. Good instinct. The beta had never told him what launch would do, because the beta was a different order of magnitude.

The question worth asking before any launch is simple. What happens at a hundred times the requests?

Small numbers hide the ceiling

A pilot with a handful of users generates a handful of model calls per minute. At that rate you never touch a limit. The provider serves you without complaint. The monthly bill is a rounding error. Everything you observe tells you the feature is cheap and reliable, because at that volume it is.

Now multiply the audience. A public launch, a feature turned on for every account, a spike from a campaign. Suddenly the same feature is making orders of magnitude more calls. Two ceilings you never felt in the beta appear at once.

The first is the provider rate limit. Model APIs cap how many requests and how many tokens you can send per minute. In the beta you lived far under that cap. At launch volume you hit it, and when you hit it the provider starts rejecting or throttling your calls. Your feature does not slow down gracefully. It starts returning errors to real users during the busiest moment you will ever have.

The second is cost. A call that costs a fraction of a cent is invisible at beta volume. At launch volume, multiplied across every user and every action, that fraction becomes a real line on the bill. Teams that never modeled it get a finance conversation they did not expect, sometimes mid-launch, sometimes with a decision to make about turning the feature off.

Model the real volume before launch

The lesson is to do the arithmetic before the traffic does it for you. Estimate the real audience, estimate calls per user per session, and multiply. Compare that number against your provider’s published limits and against a cost ceiling you can actually afford. Do this on paper first, then test against it.

Teams that launch smoothly tend to prepare in a few ordinary ways.

They estimate peak calls per minute at full audience, not average calls per day.
They request rate-limit headroom from the provider ahead of the launch, not during it.
They add queueing or backpressure so a spike degrades into a short wait instead of a wall of errors.
They set a cost ceiling and a way to see spend in near real time, so a runaway bill is visible on day one.
They cache and batch where the workload allows, because the cheapest call is the one you did not repeat.

This is not glamorous work. It is a spreadsheet and a load test. It is also the difference between a launch that holds and a launch that melts at the exact moment the most people are watching.

How we approach it at Density Labs

Most AI features that fail at launch do not fail on the model. They fail on volume that nobody modeled, hitting a rate limit or a cost ceiling that the pilot was too small to reveal. The pilot was never a scaled-down launch. It was a different system.

Our AI Opportunity Assessment is a fixed two-week engagement, priced at $2,500, that scopes the real production work before you build. Part of that is modeling the request volume of your actual audience and checking it against provider limits and a cost budget, so launch day is something you rehearsed rather than something that happens to you.

Do the multiplication before your users do.