Definition

Inter-annotator agreement

Inter-annotator agreement (IAA) measures how consistently independent human judges label or rate the same items, using statistics like percent agreement, Cohen’s kappa, or Krippendorff’s alpha. In preference data collection it is the primary quality signal: low agreement flags ambiguous tasks, noisy judges, or genuinely subjective items.

Reading agreement numbers

Raw percent agreement flatters easy tasks (two judges agreeing 80% on a coin-flip-hard task is signal; on an obvious task it is noise), which is why chance-corrected statistics exist. For pairwise preference tasks, practical shorthand works: items where 8+ of 10 votes align are strong training pairs; items splitting 5–5 are either genuinely ambiguous or badly posed. Filter them from DPO training sets, but note that for taste-driven products a true 50/50 split is itself information.

Agreement as a pipeline gate

Mature preference pipelines use agreement twice: per-annotator (a judge who disagrees with consensus far more than peers is drifting or clicking through) and per-item (low-agreement items get more votes, rewritten instructions, or exclusion). Collecting 5–10 independent votes per pair is the standard that makes both measurements possible.