The frequency test: how often does this decision actually happen
An operations director wanted AI to handle a tricky routing call. Then he counted how often it came up. The answer changed the whole project.
The frequency test: how often does this decision actually happen
An operations director at a freight brokerage described the feature he was sure he wanted. When a load got stuck between two carriers with conflicting availability, someone on his team had to decide which one to reassign it to. The call was fiddly. It pulled in rates, transit windows, and a carrier’s recent reliability. He wanted a model to make the recommendation so his people could stop stopping their day for it.
Reasonable request. Real pain. We asked him one question before anything else. How often does this actually happen.
He went to find out. The honest answer was about twice a week.
Twice a week is not a machine’s job
A decision that shows up twice a week does not add up to enough repetition to earn an AI feature. Think about what the feature costs. You scope it, you build it, you connect it to the rate data and the carrier history, you write the evals, you watch it in production, and you keep it working as those upstream systems change. That cost is mostly fixed. It does not shrink because the decision is rare. So a rare decision spreads the same build over a tiny number of uses, and the payback never arrives.
There is a second cost that hits low-frequency features harder. Trust. A person who makes this call twice a week will not build a habit around a tool they touch that seldom. Every time they come back to it, they half remember how it behaves, they second-guess it, and they end up redoing the reasoning by hand anyway. The feature does not remove the work. It adds a step in front of the work.
Frequency is the gate, not the last check
Most teams put frequency near the end of scoping, if they put it anywhere. They fall for the shape of the problem. This decision is annoying, it is a little complex, it feels like exactly the sort of judgment a model could learn. All true, and all beside the point if it happens twice a week. The interesting-ness of a decision and the value of automating it are two different measurements, and people confuse them constantly.
So run frequency first. Before you talk about accuracy, before you talk about data, ask how many times a week or a day this decision gets made across everyone who makes it. Count it honestly, in the messy real world, not in the tidy version where you imagine it happening more than it does. A good rule of thumb: if you cannot point to the decision happening many times a day somewhere in the business, the AI version has to clear a much higher bar to be worth building.
A product lead at a logistics platform put the same idea to me from the other side. Her most valuable AI feature was not the clever one. It was a boring classification that fired thousands of times a day across every shipment. It was not smart. It was constant. That constancy is what turned a small per-use benefit into something that mattered. High frequency forgives a modest benefit. Low frequency punishes even a large one.
What to do with the rare-but-real decision
The freight director did not have a nothing problem. He had a real one that failed the frequency test. That is a specific situation with specific answers, and none of them is “build the model anyway.”
Sometimes the right move is a checklist or a simple rule that a human runs in two minutes, no model required. Sometimes it is a shared document that captures how the best person on the team makes the call, so anyone can make it well. Sometimes the decision is rare because it is a symptom of something upstream, and fixing the upstream thing makes it disappear. And sometimes you group it with a family of similar low-frequency decisions until, together, they happen often enough to justify one feature that handles all of them.
What you do not do is spend a real build on a twice-a-week event and call it AI strategy.
How we approach it at Density Labs
Frequency is one of the first things we count in the AI Opportunity Assessment, our fixed two-week engagement at $2,500 that scopes the real work before anyone writes code. We ask you to show us the decision happening, not describe it, and we count how often it truly fires. Plenty of good-sounding features do not survive that count, and finding out in week one is a lot cheaper than finding out after a quarter of building.
Before you scope how smart the feature should be, count how often it runs. Rare decisions almost never pay back the machine you build to make them.