Definition
RLHF (Reinforcement Learning from Human Feedback)
RLHF (Reinforcement Learning from Human Feedback) is a training method that aligns AI models with human judgment: humans compare pairs of model outputs, a reward model learns to predict those preferences, and the base model is then optimized, typically with PPO, to produce outputs the reward model scores highly.
The data RLHF actually needs
Everything downstream of RLHF rests on one dataset: pairwise human preferences. Each training example is a prompt with a chosen and a rejected response. The reward model is only as good as those pairs: their volume, their quality, and crucially whose preferences they encode. Preferences from a narrow annotator pool bake that pool's taste into the model.
This is why the sourcing question matters as much as the algorithm. Preferences collected from representative end users produce reward signals aligned with the market the model serves; preferences from professional labelers optimize for rubric compliance.
RLHF vs DPO
DPO (Direct Preference Optimization) consumes the identical chosen/rejected pairs but skips the explicit reward model, optimizing the policy on preferences directly. RLHF offers more control (the reward model can be probed, reused, and monitored for reward hacking); DPO is simpler and cheaper to run. The collection pipeline is the same either way, which is why teams pick the algorithm late and the data source early.
Common questions
InstructGPT used tens of thousands of comparisons; modern open-source reward models train on hundreds of thousands of pairs, while narrow-domain applications succeed with a few thousand. Pilot small, measure inter-annotator agreement, then scale what works.
Related terms