Definition

Reward model

A reward model is a neural network trained on human preference data to predict how much a person would like a given AI output. It scores candidate responses during RLHF training and inference-time ranking, standing in for a human judge so that preference signal collected once can be applied to millions of generations.

Training a reward model

The standard recipe is Bradley–Terry-style training on pairwise comparisons: for each prompt with a chosen and rejected response, the model learns to score the chosen one higher. Datasets range from tens of thousands to millions of pairs; quality and diversity of the underlying human judgments dominate architecture choices in practice.

Reward hacking and drift

A policy optimized against a fixed reward model eventually finds its blind spots: outputs the reward model scores highly but humans dislike. This is reward hacking, and the defense is periodic re-grounding: collect fresh human preferences on the current policy's outputs, retrain or recalibrate the reward model, repeat. A standing pipeline for on-demand human preference collection turns this from a quarterly fire drill into routine hygiene.