Load testing an AI feature when every request costs money

A backend lead at a fintech tool told me they had load tested everything except the AI feature. For their normal services, load testing was routine: point a tool at the endpoint, send a flood of requests, watch where it bends. They did it before every launch. For the AI feature, they did not, and when I asked why, the answer was honest. Every request to the model cost money. Sending a flood of them meant sending a flood of dollars, and nobody wanted to explain that bill. So they launched without load testing the one part that behaved least like the rest.

At launch, the feature hit real traffic for the first time. The model’s rate limits, which no test had ever pushed against, started rejecting calls. Latency under concurrency, which no test had measured, was worse than anyone assumed. The cost per request at real volume, which existed only as a guess, turned out to be a guess. All of it was discoverable before launch. None of it had been tested, because the usual way to test it was too expensive to run.

The tool they trusted did not fit the target. So they aimed it at everything else and left the model untested, which is the worst version of the choice.

The old load test does not fit a metered dependency

A stateless service is cheap to hammer. Ten thousand requests cost you some compute you already own, so the natural load test is brute force. A paid model breaks that assumption. Ten thousand real requests cost ten thousand requests’ worth of money, and the whole point of the test is to send a lot of them. The technique that works for free services is priced out for a metered one, and teams respond by not testing at all rather than testing differently.

That leaves three things unknown at launch, and they are exactly the three that bite. How the feature behaves as concurrency rises. How close real load runs to the provider’s rate limits. What the feature actually costs per request when volume is real instead of estimated. Skipping the test does not remove those risks. It moves the moment you learn about them from a controlled test to an uncontrolled launch, which is a strictly worse time to find out.

What good teams do

Test with sampled traffic. You do not need full volume to learn the shape; a representative sample against the real model tells you most of it, at a bounded cost.
Mock the model for the parts that are not about the model. To test your queueing, backpressure, and concurrency handling, a stand-in that mimics the model’s latency is enough and free.
Use a cheaper model tier for the load-shape test, then price separately against the real tier on a smaller run.
Cap the spend. Run cost-bounded tests with a hard ceiling, so a load test can never become a runaway bill.
Test the limits on purpose. Push toward the provider’s rate limit deliberately in a controlled run, so launch is not the first time you meet it.

How we approach it at Density Labs

Our AI Opportunity Assessment is a fixed two-week engagement at $2,500 that scopes real production work before you build. Load testing an AI feature is one of the gaps we look for, because the reason it is skipped is so understandable. We ask how the feature was tested under load, and the answer is often that it was not, because the normal method was too expensive. Then we lay out a method that is not: sampled real traffic, mocked tiers for the parts that are about your system rather than the model, and a hard cost cap on every run.

The goal is to move the surprises before launch, where they are cheap, instead of after, where they arrive as a rejected-call incident and an unexpected invoice on the same day.

A metered model needs a metered test. Sample it, mock what is not about the model, cap the spend, and stop letting launch be your first load test.