Human preference data API
The human preference data API for RLHF, DPO, and evals
Send image, video, audio, or text outputs from your models. Get back chosen and rejected pairs, rankings, and ratings from real people, with agreement scores and every vote behind them. 10,000+ judgments a minute from 1B+ people in 200+ countries.
A 30-minute call to scope your study. A panel spec and a price follow within a working day.
Thinking Machines
One request in. Training data out.
Every job returns two layers: a verdict per datapoint to train on, and the raw votes behind it for audits or your own aggregation.
GET /jobs/{id}/resultsOne verdict per datapoint, with a reliability-weighted version beside the raw counts.
{
"datapoint_index": 0,
"votes": { "A": 7, "B": 3 },
"total_responses": 10,
"consensus": "A",
"confidence": 0.7,
"agreement_rate": 0.7,
"weighted_votes": { "A": 6.1, "B": 2.4 },
"weighted_consensus": "A",
"weighted_confidence": 0.718
}From DPO pairs to rubric scores.
Each goal maps to a task type. Score several criteria in one job, such as realism and motion, and each gets its own verdict plus a weighted composite.
| Goal | Task type | What you get |
|---|---|---|
| DPO and IPO | Comparison, or a two-item ranking for text | Chosen and rejected pairs, each with a vote margin and confidence |
| Reward models | Comparison or ranking | Pairwise outcomes for a Bradley-Terry loss. A ranking yields every implied pair. |
| Online RL | Comparison or rating | Human rewards, read back while the policy trains |
| Evals and leaderboards | Comparison | Votes per pair, ready to fit Elo or Bradley-Terry ratings |
| Rubric scoring and data filtering | Rating or multiple choice | Scores and labels for grading outputs or filtering training data |
| Critiques | Text answer | Written feedback in the rater’s own words |
Comparisons take image, video, or audio pairs, and image-to-video pairs are judged against a reference image. Text outputs, such as LLM responses, are judged with a two-item ranking or a rating.
Every label ships with the evidence behind it.
These checks run on every job automatically, and every verdict reports how strongly its raters agreed.
- Identity
- Every rater is verified as a real, unique person before their first task.
- Behaviour
- Timing and answer patterns flag bots and speed-clickers as they answer.
- Attention
- Embedded attention checks catch raters who skim instead of reading.
- Honeypots
- Hidden known-answer items drop any rater who gets them wrong.
- Bias
- Candidates appear in a shuffled order for each rater. Straight-lining and position bias are caught and removed.
- Reliability scores
- Each rater carries a platform-computed reliability score, and every verdict also comes weighted by it.
- confidence
- c = (1/n) · Σ 1[r_i = ŷ]
- weighted_confidence
- c_w = Σ w_i · 1[r_i = ŷ] / Σ w_i
ŷ is the winning answer and w_i is rater i’s reliability score. In the sample result, 7 of 10 votes pick A, so c = 0.7. Weighted, A holds 6.1 of 8.5, so c_w = 0.718.
Judged by the people your model serves.
Tasks reach 1B+ people in 200+ countries through an in-app ad network, so judgments come from everyday consumers, not career labelers.
Target directly
Country, region, city, or postal area. US area income. Gender and age range. Every result breaks down by country, and by gender or age when you target them.
Screen for anything else
Add a qualifying question, for language, profession, or product use, that ends the task for raters who don’t match. Screened-out raters are never billed.
The data behind DP-Bench, published.
The complete vote records behind the DP-Bench image, video, and speech leaderboards are on Hugging Face under CC-BY-4.0.
2M+ votes · 30 models
text-2-image-human-preferences-2m
The record behind Image Bench: a full round-robin of 30 text-to-image models on 500 prompts, 10 votes per pair, each vote carrying the rater’s trust score.
326K votes · 15 models
text-to-video-human-preferences-326k
The record behind Video Bench: 15 text-to-video models on 314 prompts built to stress motion and physics, 10 votes per pair, with every clip included.
315K votes · 15 models
text-to-speech-human-preferences-315k
The record behind Audio Bench: a full round-robin of 15 text-to-speech models over 300 English prompts in eight voice-agent categories.
Pay per response collected.
Billing is per response collected, not per job created. Volume pricing, invoicing, and enterprise terms are set on a call.
- Each job reserves its maximum cost up front and releases whatever it doesn’t use.
- Targeting adds a per-response surcharge, fixed when the job is created. A quote endpoint prices it before you launch.
- Screened-out raters and sandbox dry-runs cost nothing.
- New accounts get signup credits to test with.
Common questions.
An API that collects human judgments of model outputs on demand: which of two outputs is better, how several outputs rank, or how one output scores. With Datapoint you send outputs in a request and get back verdicts with agreement scores plus every rater’s vote, ready for RLHF, DPO, reward models, or evaluation.
Run a comparison job over pairs of outputs from your model, or a two-item ranking for text. Each result names the preferred output with its vote margin and confidence, so you can keep only the pairs above an agreement threshold as chosen and rejected examples.
Both use pairwise preferences. RLHF trains a separate reward model on them and then optimizes the policy against it, while DPO optimizes the policy on the pairs directly. The format is the same, chosen and rejected pairs, so one job can feed either pipeline.
It depends on the domain. Narrow DPO fine-tunes often start from a few thousand pairs, while general-purpose reward models use far more. A small pilot on your own outputs measures rater agreement first, and the call is where we size the full run.
Each datapoint collects 10 judgments by default, and you can set a different count per job. Close calls need more votes to separate, so raise the count for evals between similar models, and use the reported confidence to drop ambiguous pairs from training sets.
Every rater is verified as a real, unique person, and answers pass timing, attention, known-answer, and bias checks before they count. Every verdict also comes in a reliability-weighted form that weights each rater by a platform-computed reliability score.
Yes. Filter by country, region, city, postal area, US area income, gender, or age range, and add a screening question for anything else, such as language or profession. Raters who are screened out are never billed.
First responses typically arrive within minutes, most datapoints reach their full vote count within about an hour, and a job with hundreds of datapoints usually finishes in a few hours. Partial results can be read at any point while a job runs.
You pay per response collected. Targeting adds a per-response surcharge that is fixed when the job is created, and sandbox dry-runs are free. Volume pricing and enterprise terms are agreed on a call.
Yes. The Datapoint MCP server lets agents in Claude Code, Cursor, Codex, and other MCP clients plan, launch, and read preference studies through the same API.
Yes. The complete vote records behind the DP-Bench image, video, and speech leaderboards are on Hugging Face under CC-BY-4.0, including 2M+ text-to-image votes across 30 models.
Scope your preference data on a 30-minute call.
Bring the modality, the volume, and the population you need judged. A panel spec and a price follow within a working day.