For AI & research teams
The human preference data API.
Pairwise comparisons, ratings, and rankings from verified real people, collected through one API call and streamed back in minutes. Built for RLHF and DPO training pairs, reward models, and human evals of generative models across text, image, audio, and video.
One POST to launch. Votes stream back in minutes.
Describe the judgment you need in plain language, attach your model outputs, and set how many responses each datapoint should collect. Datapoint recruits the humans, runs the quality gates, and returns aggregated votes. No panels to manage, no annotation workforce to contract.
Create a comparison job
curl -X POST https://api.trydatapoint.com/data-labelling/v1/jobs \
-H "X-API-Key: $DATAPOINT_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "sdxl-vs-flux",
"instruction": "Which image better matches the prompt?",
"task_type": "comparison",
"max_responses_per_datapoint": 10,
"datapoints": [{
"media": {
"candidates": [
{ "url": "dp://b6a1cd3f1234/gen_a.png", "type": "image" },
{ "url": "dp://c8f3ae9b5678/gen_b.png", "type": "image" }
]
}
}]
}'Every preference format your training pipeline expects.
Pairwise comparison
Show two model outputs side by side and ask which one wins. The canonical format for RLHF and DPO preference pairs, and for Elo-style model rankings.
Rating
Score a single output on a numeric scale: quality, realism, prompt faithfulness, or any rubric you define in plain language.
Ranking
Order three or more candidates best-to-worst. One judgment yields multiple implied pairwise preferences.
Multiple choice
Structured categorical questions: which artifact is present, which caption fits, which response follows the instruction.
Across
TextImagesAudioVideoPreferences from the people your model serves, not a labeling workforce.
Preference data is only as good as the humans behind it. Professional annotators develop rubric habits and stop resembling real users; panels skew toward whoever signed up years ago. Datapoint's respondents are everyday consumers recruited live through in-app ads, so the taste you train on is the taste of the market.
Real consumers, not professional annotators
Datapoint recruits respondents through a global in-app ad network, so judgments come from fresh, everyday people (the population your model actually serves), not from career labelers who have internalized rubric-gaming habits and drifted from typical user taste.
Demographic reach you can target
Respondents span 200+ countries with verified device-level identity. Target by geography, age band, or interest when your preference data needs to reflect a specific population rather than whoever is on a labeling marketplace today.
Quality gates on every response
Attention checks, response-time floors, agreement scoring, and device verification filter every stream. You get the cleaned votes plus per-datapoint agreement stats, not raw clicks.
Millions of preference votes, already collected with this API.
2M+ votes
text-2-image-human-preferences-2m
Pairwise human preferences across leading text-to-image models, collected through this API and published openly on Hugging Face.
145K judgments
VidPrefMotion
Human preference judgments on text-to-video generations, focused on motion quality. This is the dataset behind our video model benchmark.
RLHF · DPO · Reward models
From raw generations to training-ready pairs.
- Upload candidate outputs, get chosen/rejected pairs with vote counts and agreement scores, ready for DPO fine-tuning or reward model training.
- Run periodic human evals against your automated metrics to catch reward hacking and judge drift before they compound.
- Scale from a 1K-pair pilot to hundreds of thousands of judgments on the same job definition, paying per response.
Frequently asked
Human preference data records which of two or more options real people prefer, typically pairwise comparisons, ratings, or rankings of model outputs. It is the core training signal for RLHF and DPO, the ground truth for reward models, and the standard way to evaluate generative models where automated metrics fall short.
With Datapoint you POST a job with your instruction, task type, and media candidates; verified respondents recruited through a global ad network answer within seconds, and you fetch aggregated votes with per-datapoint agreement stats. One curl call to launch, one to collect, with no panel management and no annotation ops.
Published fine-tunes range from a few thousand preference pairs for narrow-domain DPO to hundreds of thousands for general-purpose reward models. Because Datapoint is pay-per-response, teams usually start with a pilot of 1–5K pairs to validate their setup, measure inter-annotator agreement, then scale the same job definition.
Traditional providers route your tasks to trained, full-time labelers. Datapoint recruits everyday consumers through in-app ads, which produces preference judgments that better reflect real end-user taste, avoids annotator-population bias, and scales elastically: 100K+ people are reachable at any moment without contracting a workforce.
Pairwise comparison, rating, ranking, multiple choice, and multi-step chains, over text, images, audio, and video. You can mix media within a job, target demographics, and set how many responses each datapoint needs.
Responses start streaming within seconds of launching a job (the median individual response lands in about 15 seconds), and most studies complete within minutes, because respondents are recruited live from an ad network rather than scheduled from a panel.
Your first thousand preference pairs, this afternoon.
New accounts get free credits, enough to pilot a real study. Or book a call and we'll help design your collection.