# Pairwise comparison data collection

> Collect pairwise comparison data at scale via API: A/B judgments from verified real people on any media (text, image, audio, video) with agreement statistics, back in minutes.

The most reliable subjective judgment humans make is "which of these two is better". Collect it programmatically on any media (model outputs, ad creatives, product designs) with 5–10 independent votes per pair and agreement statistics built in.

## What teams collect

### Model output pairs
The RLHF/DPO workhorse: chosen/rejected pairs on text, image, audio, or video outputs, filtered by agreement threshold for training quality.

### Creative and design A/Bs
Ad variants, packaging, logos, landing pages: pairwise preference from your target demographic before anything ships or spends.

### Ranking via pairs
Sparse pairwise sampling across many candidates, aggregated with Bradley–Terry or Elo into a full ranking without judging every combination.

### Multi-step chains
Sequential tasks that branch on earlier answers: compare, then ask why, capturing the reason behind the preference alongside the vote.

## FAQ

### Why pairwise instead of rating scales?
People are far more consistent at relative judgments than absolute ones: rating scales suffer calibration drift between and within judges, while A/B choices cluster. If you need a global ranking, pairwise votes plus an Elo or Bradley–Terry model recover it more reliably than averaged ratings.

### How many pairs and votes do I need?
For a ranking over n candidates, sparse sampling of a fraction of all n(n−1)/2 pairs suffices. Per pair, 5–10 independent votes lets you compute agreement and filter noise; a 1–5K pair pilot is the usual starting point before scaling.

### What does it cost?
Datapoint is pay-per-response with no seats or contracts; a comparison response costs a handful of credits, and new accounts get free credits sufficient for a real pilot study.

---

All of this runs on the Datapoint human preference data API: https://trydatapoint.com/human-preference-data-api (docs: https://trydatapoint.com/docs).
