Grading the grader: when your automated judge disagrees with your users
Teams reach for an LLM to score their AI outputs at scale, then trust the judge without ever checking it against a human. The judge is a model too, with its own failures.
Grading the grader: when your automated judge disagrees with your users
Once you have more outputs than a person can read, you reach for a model to grade them. An LLM scores each answer for quality, and suddenly you can evaluate thousands of cases a night. The scores look authoritative. Then a user complains about an output your judge marked as good, and you realize you have been trusting a grader nobody graded.
The judge is a model too
A fintech CTO described the moment it clicked for his team. Their automated judge was rating a support-answer feature, and the dashboards looked healthy. Customer feedback did not match. When they pulled the disagreements and read them by hand, the judge was rewarding answers that sounded confident and fluent while missing that some were plain wrong. The grader had a taste for tone. Its scores had drifted away from what a user actually cared about.
The judge is not an oracle. It is another non-deterministic model with its own blind spots, and its errors are more dangerous than a single feature’s because they hide inside your evaluation, the one place you go to find out whether things are working.
Two habits keep the judge honest
The first is boring and non-negotiable. You need a set of cases a human has scored, and you check the judge against them the way you check any classifier. Where the judge and the human agree, you have earned some trust. Where they split, you read the case and learn whether the judge or the label is wrong. A founder who runs an AI data-security company treats this human-labeled set as the thing that grades the grader, refreshed often enough that it keeps up with the product.
The second habit deals with the judge’s own randomness. Ask the same judge the same question twice and you can get two answers. The fix is self-consistency. Sample the judge several times on a case and take the most frequent verdict instead of trusting a single roll. It costs more calls, and it turns a jittery score into a stable one.
How to run an automated judge you can trust
- Hold a human-labeled set, the ground truth you measure the judge against, not the other way around.
- Track judge-human agreement, and treat a falling agreement rate as a broken instrument.
- Sample the judge several times and take the majority verdict, because one call is not a measurement.
- Read the disagreements, since the cases where judge and human split are where you learn what the judge is bad at.
When to escalate to a person
An automated judge is for volume, not for the cases that matter most. On the high-stakes slice, the outputs where a wrong grade costs real money or trust, keep a person in the loop and use the judge to triage what reaches them. The model tells you where to look. The human decides the calls that count.
A judge you never audit is not saving you evaluation work. It is moving your errors somewhere darker, into the numbers you use to sleep at night. Grade the grader before you believe it.