Human preference data

Generative models are trained to produce outputs humans like, but "what humans like" is not written down anywhere, so it has to be measured. Preference data is that measurement: put two model outputs in front of a person, ask which is better, repeat at scale, and you have a signal that can train a reward model, fine-tune a policy with DPO, or settle which of two model versions actually improved.

The most common formats are pairwise comparisons (choose A or B), ratings (score one output on a scale), and rankings (order several outputs). Pairwise dominates because people are far more consistent at relative judgments than absolute ones, and because chosen/rejected pairs are exactly what RLHF and DPO pipelines consume.

Historically, preference data came from professional annotation workforces (Scale, Surge) or research panels (Prolific). The newer model is API-based collection from everyday consumers: you submit model outputs programmatically, real people recruited through ad networks judge them within seconds, and aggregated votes stream back with agreement statistics. This trades the consistency of trained labelers for the representativeness of actual end users, a deliberate choice when the goal is modeling market taste rather than expert rubric compliance.

Quality control matters in either model: attention checks, response-time floors, inter-annotator agreement scoring, and identity verification are standard filters before a vote counts.

Published results range from a few thousand pairs for narrow-domain DPO fine-tunes to hundreds of thousands for general-purpose reward models. The practical approach is to pilot 1–5K pairs, measure inter-annotator agreement to validate the task design, then scale the same collection setup.