What Is Inter-Annotator Agreement (IAA) — and Why Your Model Depends On It

When AI teams talk about training data quality, they usually mean accuracy — did the annotator label the image correctly? But accuracy is only half the picture. The other half is consistency: if you gave the same image to three different annotators, would they all label it the same way?

That's what inter-annotator agreement measures. And for most AI teams, it's a more actionable signal than accuracy alone — because inconsistency in labeled data is harder to detect, harder to correct, and more damaging to model performance than a predictable error rate.

What Is Inter-Annotator Agreement?

Inter-annotator agreement (IAA) — sometimes called inter-rater reliability — is a metric that measures how consistently two or more annotators label the same data. It answers the question: when different people apply our taxonomy to the same input, do they reach the same output?

IAA is typically measured by having multiple annotators label the same subset of data independently, then comparing their outputs. The higher the agreement, the more reliably the taxonomy is being applied — and the more trustworthy the full labeled dataset is as training signal.

How IAA Is Measured

There's no single IAA metric — the right one depends on your task type. Here are the most commonly used:

Cohen's Kappa (κ)

The most widely used IAA metric for classification tasks. Cohen's Kappa measures agreement between two annotators while correcting for the agreement you'd expect by chance alone. A raw agreement of 90% sounds strong — but if your classes are heavily imbalanced, 80% of that agreement might be due to chance. Kappa adjusts for this.

Formula: κ = (Po − Pe) / (1 − Pe), where Po is observed agreement and Pe is expected chance agreement.

Fleiss' Kappa

An extension of Cohen's Kappa for three or more annotators. Used when you're running consensus annotation or auditing a larger team's consistency across a project.

Krippendorff's Alpha

More flexible than Kappa — handles missing data, works across ordinal, interval, and ratio scales, and is often preferred in academic NLP research. Increasingly used in complex annotation tasks with multiple label dimensions.

IoU-Based Agreement (Computer Vision)

For bounding box, segmentation, and spatial annotation tasks, categorical agreement metrics don't apply. Instead, Intersection over Union (IoU) measures how much two annotators' bounding boxes overlap. Average Precision (AP) at different IoU thresholds (e.g., AP@0.5, AP@0.75) is the standard for evaluating consistency in object detection annotation.

What Do IAA Scores Actually Mean?

Using Cohen's Kappa as a reference point, here's how to interpret scores:

These thresholds shift by domain. For safety-critical applications — medical imaging, autonomous systems — you should target 0.85+ Kappa even for tasks that seem straightforward. For complex semantic tasks like sentiment or intent classification, 0.70 may be a reasonable ceiling depending on label granularity.

Why Low IAA Is Worse Than High Error Rate

Here's a counterintuitive truth: a dataset where annotators are consistently wrong is easier to train on than one where annotators are unpredictably inconsistent.

If your annotators consistently mislabel one edge case the same way, your model learns a clear (if incorrect) signal. You can identify the pattern, correct the labels in bulk, and retrain. But if different annotators handle the same edge case differently — one labels it class A, another labels it class B — your model receives contradictory signal. It can't learn a consistent rule, so it learns noise.

This is why IAA is a leading indicator of model performance problems. By the time low IAA shows up in your eval metrics, you've already wasted training cycles on bad data.

What to Ask Your Annotation Vendor About IAA

Strong annotation vendors track IAA continuously and proactively share it. Here's what to ask:

• What IAA metric do you use, and why? The answer should match your task type — Kappa for classification, IoU for vision tasks, Alpha for complex multi-label tasks.
• What is your baseline IAA score for tasks similar to ours? Push for domain-specific benchmarks, not a single lifetime number.
• How often do you measure IAA, and what triggers a re-review? Weekly per-sprint measurement with defined drift thresholds is the baseline expectation.
• Do you include IAA scores in delivery reports? If the answer is no, you have no visibility into quality until your model eval — which is too late.

Any vendor who can't answer these questions specifically — or who doesn't include IAA data in standard delivery reports — is asking you to trust quality claims they can't verify.