# Text-to-video human preference data

> Human preference data for video generation models: motion quality, temporal consistency, and realism judgments from real people via API. Behind the 145K-judgment VidPrefMotion dataset.

Motion quality, temporal consistency, and realism are exactly the qualities automated metrics measure worst, and humans judge instantly. Collect side-by-side video preferences at scale, with the pipeline behind our 145K-judgment VidPrefMotion benchmark.

## Published proof

- VidPrefMotion: 145K human judgments (Methodology published on our blog) at https://trydatapoint.com/blog/vidprefmotion-human-preferences

## What teams collect

### Motion quality side-by-sides
Two clips for the same prompt, judged on which moves more naturally. This is the dimension where video models differ most and VLM judges correlate with humans least.

### Temporal consistency and artifacts
Flicker, morphing objects, identity drift across frames: structured multiple-choice or rating tasks that localize failure modes per model.

### Model version regression checks
Before shipping a new checkpoint, a few hundred side-by-side votes against the incumbent catches quality regressions automated evals miss.

### Instruction following for edits
For video editing models: did the edit do what was asked, and did it preserve everything else? Pairwise comparisons against the source clip.

## FAQ

### What does human evaluation of a video model cost?
You pay a flat rate per response, however long the clips are. On platforms that bill participant hours, a 45-second video comparison comes to about $214 per 1,000 responses; Datapoint is about a quarter of that. Book a call for a quote.

### Why use human judgment for video models instead of VLM judges?
Our VidPrefMotion study benchmarked VLM judges against 145K human judgments and found substantial disagreement on motion quality, the core competency of video generation. VLM judges work as a pre-filter; human preferences remain the ground truth for launch decisions and training data.

### What video formats does the API accept?
MP4, WebM, and MOV uploads up to 20 MiB per file, or public HTTPS URLs. Respondents watch both candidates before voting; response-time floors ensure the clips were actually viewed.

### How fast can I evaluate a new video model?
A few hundred side-by-side judgments typically complete within minutes of launching the job, fast enough to run human evals inside a training loop or a release pipeline, not just quarterly.

---

All of this runs on the Datapoint human preference data API: https://trydatapoint.com/human-preference-data-api (docs: https://trydatapoint.com/docs).
