CONSENSUS AT SCALE
10 to 15 raters per item
Majority votes and full response distributions reported, with inter-annotator agreement per label and both unweighted and trust-weighted consensus confidence.
←Datapoint
For AI labs and research teams
5,000+ human preferences a minute
Pairwise preferences, rankings, and ratings from real people, served through our own mobile distribution to 9M+ annotators. Enable RLHF, refine model behaviour, and evaluate outputs with data that ships with trust scores and raw per-rater records, usually in under an hour.
How it works
Own distribution, real people, structured results.
Our own distribution
Tasks are served through the Datapoint SDK into mobile ad inventory, a network built on consumer apps with over 9M users. No panel, no marketplace, no bots.
Real people answer
Annotators complete short structured judgments in-app: pairwise preference, ranking, label verification, and capture grading across image, video, audio, and text.
Structured results
Responses stream back via API with per-rater records, trust scores, demographics, and a geographic breakdown, so the data drops straight into your training or eval pipeline.
Auditable by design
Every study ships with raw per-rater responses. Recompute agreement, run your own exclusions, and check every aggregate we report. Our public datasets are the standing audit trail.
Why the data is trustworthy
Consensus, calibration, and trust scores on every label.
CALIBRATION
Known-answer tasks in every stream
Skill-based calibration items are seeded into every task stream. Raters who miss them are excluded and their responses removed from results.
TRUST SCORES
Longitudinal rater quality
Every rater carries a quality score across studies. Low-reputation responses are filtered before aggregation, alongside response-time floors, straight-lining detection, and device deduplication.
Open data
Inspect the data before you spend a credit.
Datapoint publishes open human preference datasets on Hugging Face under CC-BY-4.0, spanning text-to-image, text-to-video, image-to-video, and speech. Every release includes raw per-item responses, so you can recompute agreement and check our aggregates yourself.
Frequently
Asked Questions
Pairwise preference, ranking, ratings, label verification, and capture grading across image, video, audio, and text. Chains let you ask follow-up questions conditioned on an earlier answer, and the annotator UI is fully customizable.
Stimulus order and screen position are shuffled per rater, and the controls are documented in our dataset cards. Screening questions, geo verification, and qualified incidence are reported on every study, so you know exactly who answered.
Yes. Our public datasets on Hugging Face are released under CC-BY-4.0 with raw per-item human responses, so you can check rater agreement and label quality before you run a study.
Pay per response, with free credits on new accounts. Academic researchers can apply to the research grant program for funded data.
Human preferences at the speed of your training loop.
Sign up, take the free credits, run your first preference study today.