When two labelers disagree, your metric is lying

Hand the same 200 examples to two careful people. They agree on 156 of them and split on 44. Most teams never run this check. The ones who do get a shock, because that 22% disagreement is the real error bar on every accuracy number they have ever reported.

Disagreement is a definition problem

When two reasonable people label the same example differently, the usual instinct is to call one of them wrong and move on. That instinct is the mistake. If both are competent and both are following the same instructions, the disagreement is not about them. It is about the task. The definition left room for two readings, and each person picked one.

That gap sets a hard ceiling on what any metric can mean. If humans only agree with each other 78% of the time, a model that scores 78% might be perfect, and a model that scores 82% might just be memorizing one labeler’s quirks. You cannot tell. The number you report to the room is noise dressed as signal, and the decimal places make it look more trustworthy than it is.

The reliability instinct applies here

A product manager at a data-pipeline company gave me a triage rule I keep reusing. Her product moves data out of fragmented SaaS apps into a warehouse for AI and analytics, and her whole job is deciding what actually threatens the customer. Her line: if the reliability target is met you can defer the small stragglers, but if something “leads to suboptimal customer benefit, you’ve built the ATM and money doesn’t come out, you fix that technical debt ASAP.”

Label disagreement is an ATM that will not dispense. The interface looks fine. The accuracy dashboard renders a clean number. Underneath, the definition is broken, so the metric it produces does not connect to reality. That is the kind of debt you fix now, not the kind you defer, because everything you decide downstream leans on it.

Fix the guideline, then the model

The reflex when accuracy is low is to change the model. If your inter-annotator agreement is weak, that reflex burns weeks. You will tune toward one interpretation of an ambiguous rule and call it progress, while a second labeler would score the same outputs completely differently.

The order that works is the reverse. Measure agreement first. Find the examples where careful people split. Read them together and ask what the guideline failed to say. Usually there is a specific edge the instructions never addressed, a case that could honestly go either way. Write the rule that resolves it, relabel the affected examples, and only then look at the model. You have fixed the ruler before measuring with it.

Say you push agreement from 78% to 94% by tightening three ambiguous rules. Now a model scoring 90% means something, because the humans it is graded against actually concur on what right looks like. The same model that looked mediocre against a noisy set looks strong against a clean one, and neither the model nor its training changed. Only the definition did.

A cheap check most teams skip

This costs one afternoon. Pull a sample, have two people label it independently, compare. Where they agree, your definition is solid. Where they split, you found the ambiguity that has been quietly corrupting your metrics, and you found it before it corrupted a launch decision. Skipping this check does not make the ambiguity go away. It just moves the discovery to production, where it costs more.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two week, $2,500 engagement, we look for whether anyone ever measured agreement between labelers. When the task is ambiguous, we surface it as a definition finding, not a model finding, because tuning against a noisy label set wastes the whole pilot. Fixing the guideline is faster, cheaper, and it makes every metric downstream mean something.

If your labelers cannot agree, your model cannot be graded. Settle the definition first.