Definition

DPO (Direct Preference Optimization)

DPO (Direct Preference Optimization) fine-tunes a language or generative model directly on human preference pairs, each a prompt with a chosen and rejected response, without training a separate reward model. It consumes the same pairwise preference data as RLHF but replaces reinforcement learning with a simpler classification-style loss.

RLHF vs DPO data requirements

The data format is identical: chosen/rejected pairs. What changes is sensitivity. Because DPO optimizes on pairs directly with no reward model to smooth over noise, label quality cuts deeper: a mislabeled pair is a gradient step in the wrong direction, not one vote among many shaping a reward function. Teams running DPO typically collect more votes per pair (5–10) and filter to high-agreement pairs before training.

Volume-wise, DPO has shown strong results from smaller datasets than classic RLHF: a few thousand high-quality, high-agreement pairs can meaningfully shift a model in a narrow domain. The budget saved on reward-model training is often better spent on cleaner preference collection.

Where the pairs come from

Any pairwise preference source works: side-by-side comparisons of two model checkpoints, two sampling temperatures, or a model against a baseline. API-based human preference collection slots in directly: generate candidate pairs, collect votes with agreement statistics, keep the pairs that clear your threshold, train.