Glossary
Human preference data, defined term by term
The vocabulary of preference collection and human evaluation, written for teams training and evaluating models, not for a textbook.
Human preference data
Human preference data records which of two or more options real people prefer, most often as pairwise comparisons, ratings, or rankings of AI model outputs. It is the core training signal for RLHF and DPO, the ground truth for reward models, and the standard for evaluating generative models where automated metrics fall short.
Pairwise comparison
A pairwise comparison presents two options side by side and asks a judge to pick the better one. In AI evaluation it is the dominant way to collect human preference data, because people make relative judgments far more consistently than absolute ratings, and the resulting chosen/rejected pairs feed RLHF, DPO, and Elo-style leaderboards directly.
RLHF (Reinforcement Learning from Human Feedback)
RLHF (Reinforcement Learning from Human Feedback) is a training method that aligns AI models with human judgment: humans compare pairs of model outputs, a reward model learns to predict those preferences, and the base model is then optimized, typically with PPO, to produce outputs the reward model scores highly.
DPO (Direct Preference Optimization)
DPO (Direct Preference Optimization) fine-tunes a language or generative model directly on human preference pairs, each a prompt with a chosen and rejected response, without training a separate reward model. It consumes the same pairwise preference data as RLHF but replaces reinforcement learning with a simpler classification-style loss.
Reward model
A reward model is a neural network trained on human preference data to predict how much a person would like a given AI output. It scores candidate responses during RLHF training and inference-time ranking, standing in for a human judge so that preference signal collected once can be applied to millions of generations.
Human-in-the-loop (HITL)
Human-in-the-loop (HITL) describes AI systems that incorporate human judgment at some stage of their pipeline: labeling training data, evaluating outputs, resolving low-confidence cases at runtime, or auditing automated decisions. The term covers everything from annotation workflows to production systems that escalate uncertain predictions to people.
LLM-as-a-judge
LLM-as-a-judge is an evaluation method where a strong language model grades or ranks the outputs of another model, replacing or supplementing human judges. It is fast and cheap at scale, but carries documented biases (position, verbosity, self-preference) and must be validated against human preference data to be trustworthy.
Human evaluation
Human evaluation measures an AI model's quality by having people judge its outputs through side-by-side comparisons, ratings, or task-success assessments, rather than relying on automated metrics. It is the gold standard for subjective qualities like helpfulness, realism, and aesthetics, and the benchmark against which automated evaluators are calibrated.
Annotator bias
Annotator bias is the systematic skew introduced into labeled or preference data by the particular people who produced it: their demographics, culture, expertise, and incentives. Models trained on biased preferences inherit them, which makes the composition of the annotator pool as consequential as the size of the dataset.
Inter-annotator agreement
Inter-annotator agreement (IAA) measures how consistently independent human judges label or rate the same items, using statistics like percent agreement, Cohen’s kappa, or Krippendorff’s alpha. In preference data collection it is the primary quality signal: low agreement flags ambiguous tasks, noisy judges, or genuinely subjective items.
Preference dataset
A preference dataset is a structured collection of human preference judgments, typically prompts paired with a chosen and a rejected model output, plus vote counts or judge metadata. Preference datasets train reward models, fuel DPO fine-tuning, and benchmark generative models against human taste.
Side-by-side evaluation
Side-by-side (SxS) evaluation presents a judge with two outputs for the same input, typically from two models or two versions of one model, and asks which better satisfies the task. It is the standard human-evaluation design for model comparisons, powering launch decisions and public leaderboards alike.
Elo rating (for AI models)
An Elo rating converts pairwise comparison outcomes into a single skill score per competitor, updating after each head-to-head result. Borrowed from chess, it is now the standard way AI leaderboards such as Chatbot Arena and generative-model benchmarks turn millions of human preference votes into a global model ranking.
Need the data these terms describe? Datapoint is an API for human preference data: pairwise comparisons, ratings, and rankings from 100K+ verified people, back in minutes.