Definition

Preference dataset

A preference dataset is a structured collection of human preference judgments, typically prompts paired with a chosen and a rejected model output, plus vote counts or judge metadata. Preference datasets train reward models, fuel DPO fine-tuning, and benchmark generative models against human taste.

Notable public preference datasets

Anthropic’s HH-RLHF (helpfulness/harmlessness pairs), OpenAssistant, LMSYS Chatbot Arena conversations, and Pick-a-Pic / ImageReward for text-to-image are widely-used public sets. Datapoint publishes its own on Hugging Face: text-2-image-human-preferences-2m (over two million pairwise votes across leading image models from everyday consumers worldwide) and VidPrefMotion, 145K judgments on text-to-video motion quality.

Building your own

Public datasets bootstrap general capability, but they encode other people’s prompts, models, and judges. Product-specific alignment needs preferences on your outputs from your target population. The recipe: generate candidate pairs from the models you care about, collect 5–10 independent human votes per pair with demographic targeting, keep agreement statistics, and version the dataset alongside the models it trained.