Offline eval and online eval measure two different things
A feature can pass every test on your fixed dataset and still fail with real users. The two evaluations answer different questions, and skipping either one leaves a blind spot.
Offline eval and online eval measure two different things
A feature can pass every case in your evaluation set and still disappoint the first users who touch it. That is not a contradiction. Offline evaluation and online evaluation answer different questions, and a team that runs only one is confident about half of what it needs to know. The offline set tells you the feature is correct on the cases you thought of. Users tell you whether it is useful on the ones you did not.
Offline eval answers “is it correct on what we know”
Offline evaluation runs the feature against a fixed set of inputs with expected outcomes, before anything ships. It is fast, it is repeatable, and it is the only kind of test you can run on every change without waiting. A developer-tools founder described leaning on it hard during build. His team could run the whole set in minutes and catch a regression the same afternoon they caused it, long before a user would have.
Its limit is built into its strength. The set is fixed, so it only ever tests the cases someone chose to include. A feature that scores perfectly offline has proven it handles the world you imagined. It has proven nothing about the world that is about to send it real inputs.
Online eval answers “is it useful in the wild”
Online evaluation happens after launch, with live traffic. It measures things the offline set structurally cannot: whether users accept the output, how often they escalate to a human, where they abandon the feature, what inputs arrive that nobody predicted. A semiconductor engineering leader made the point that the field always sends conditions the lab did not. His bench tests were necessary and never sufficient, because reality runs a wider set of inputs than any bench.
The two are a loop. Online eval surfaces the inputs you failed to imagine. Those inputs become new cases in the offline set. The offline set gets closer to reality with every surprise the live feature hands you, and the next regression gets caught before launch instead of after.
Where each one belongs
- Run offline eval on every change, as the fast gate that catches regressions before they ship.
- Run online eval continuously after launch, watching acceptance, escalation, and abandonment on real traffic.
- Feed online surprises back into the offline set, so each unpredicted input becomes a permanent test.
- Do not let a green offline score end the conversation, because it only speaks for the cases you already knew.
The blind spot each one leaves
A team that runs only offline eval ships features that are correct and unwanted, tuned to a set that reality does not match. A team that runs only online eval learns from every failure in production, which is the slowest and most public classroom there is. The offline set makes failure cheap. The online signal makes the set honest. You want the pair, each correcting the other.
Passing your own test set means you built the thing you specified. Whether you specified the right thing is a question only users can answer. Keep both instruments running.