Definition
Pairwise comparison
A pairwise comparison presents two options side by side and asks a judge to pick the better one. In AI evaluation it is the dominant way to collect human preference data, because people make relative judgments far more consistently than absolute ratings, and the resulting chosen/rejected pairs feed RLHF, DPO, and Elo-style leaderboards directly.
Pairwise vs rating scales
Ask ten people to rate an image 1–10 and you get ten different calibrations of the scale: one person's 6 is another's 8, and the same person drifts over a session. Ask the same ten people which of two images is better and their answers cluster. Decades of psychometrics (Thurstone, Bradley–Terry) and modern LLM evaluation practice agree: relative judgment is the more reliable instrument for subjective quality.
The trade-off is coverage: n outputs need n(n−1)/2 pairs for a full round-robin. In practice, sparse sampling plus a rating model (Elo, Bradley–Terry) recovers a global ranking from a small fraction of all possible pairs. This is the approach used by Chatbot Arena and most model leaderboards.
Collecting pairwise data at scale
Pairwise tasks are also the easiest judgment to crowdsource: a comparison takes seconds, requires no rubric training, and works for text, images, audio, and video. API-based platforms let you submit candidate pairs programmatically and set how many independent votes each pair collects; 5–10 votes per pair with an agreement threshold is a common recipe for training-quality data.
Related terms