# DPO (Direct Preference Optimization)

> DPO (Direct Preference Optimization) fine-tunes a language or generative model directly on human preference pairs, each a prompt with a chosen and rejected response, without training a separate reward model. It consumes the same pairwise preference data as RLHF but replaces reinforcement learning with a simpler classification-style loss.

## RLHF vs DPO data requirements

The data format is identical: chosen/rejected pairs. What changes is sensitivity. Because DPO optimizes on pairs directly with no reward model to smooth over noise, label quality cuts deeper: a mislabeled pair is a gradient step in the wrong direction, not one vote among many shaping a reward function. Teams running DPO typically collect more votes per pair (5–10) and filter to high-agreement pairs before training.

Volume-wise, DPO has shown strong results from smaller datasets than classic RLHF: a few thousand high-quality, high-agreement pairs can meaningfully shift a model in a narrow domain. The budget saved on reward-model training is often better spent on cleaner preference collection.

## Where the pairs come from

Any pairwise preference source works: side-by-side comparisons of two model checkpoints, two sampling temperatures, or a model against a baseline. API-based human preference collection slots in directly: generate candidate pairs, collect votes with agreement statistics, keep the pairs that clear your threshold, train.

## Related terms

- https://trydatapoint.com/glossary/rlhf
- https://trydatapoint.com/glossary/human-preference-data
- https://trydatapoint.com/glossary/pairwise-comparison
- https://trydatapoint.com/glossary/inter-annotator-agreement

---

Collect this kind of data with the Datapoint human preference data API: https://trydatapoint.com/human-preference-data-api
