Text-to-image · Preference data
Text-to-image human preference data
Pairwise preferences, prompt-faithfulness ratings, and aesthetic judgments on image model outputs, from everyday consumers rather than professional annotators. The same pipeline behind our 2M+ vote public dataset, available through one API.
What teams collect
Judgment tasks that work for text-to-image
Model vs model side-by-sides
Two generations for the same prompt, one vote for the better image. Aggregates into win rates and Elo rankings across model versions, checkpoints, or providers.
Prompt faithfulness
Does the image actually contain what the prompt asked for? Ratings or A/Bs focused on adherence rather than aesthetics. This is the failure mode automated metrics miss most.
Aesthetic quality and style
Which image is more beautiful, more on-brand, more like the reference style? Pure taste judgments where representative consumers are the only valid ground truth.
DPO training pairs
High-agreement chosen/rejected image pairs, filtered by vote threshold, ready for Diffusion-DPO-style fine-tuning or reward model training.
Frequently asked
You submit image pairs (or single images with a rating rubric) through the Datapoint API; verified respondents recruited via a global ad network judge them within seconds. Each datapoint collects as many independent votes as you specify, and results return with per-item agreement statistics.
Yes. The public dataset was collected with the same jobs API available to every account: pairwise comparison tasks, 5+ votes per pair, quality gates on every response. Point the same setup at your own model outputs.
For leaderboards, 5 votes per pair across many pairs beats many votes on few pairs. For DPO training data, 5–10 votes with a high-agreement filter (e.g. keep pairs where 80%+ agree) is a common recipe.
Everyday consumers across 200+ countries, recruited live through in-app ads and verified at the device level, not career annotators. You can target by geography, age, or interest when preferences should reflect a specific market.
Pilot a real text-to-image study today.
Free credits on new accounts, pay-per-response after. No contracts, no annotation ops.