Text-to-video · Preference data
Text-to-video human preference data
Motion quality, temporal consistency, and realism are exactly the qualities automated metrics measure worst, and humans judge instantly. Collect side-by-side video preferences at scale, with the pipeline behind our 145K-judgment VidPrefMotion benchmark.
What teams collect
Judgment tasks that work for text-to-video
Motion quality side-by-sides
Two clips for the same prompt, judged on which moves more naturally. This is the dimension where video models differ most and VLM judges correlate with humans least.
Temporal consistency and artifacts
Flicker, morphing objects, identity drift across frames: structured multiple-choice or rating tasks that localize failure modes per model.
Model version regression checks
Before shipping a new checkpoint, a few hundred side-by-side votes against the incumbent catches quality regressions automated evals miss.
Instruction following for edits
For video editing models: did the edit do what was asked, and did it preserve everything else? Pairwise comparisons against the source clip.
Frequently asked
Our VidPrefMotion study benchmarked VLM judges against 145K human judgments and found substantial disagreement on motion quality, the core competency of video generation. VLM judges work as a pre-filter; human preferences remain the ground truth for launch decisions and training data.
MP4, WebM, and MOV uploads up to 20 MiB per file, or public HTTPS URLs. Respondents watch both candidates before voting; response-time floors ensure the clips were actually viewed.
A few hundred side-by-side judgments typically complete within minutes of launching the job, fast enough to run human evals inside a training loop or a release pipeline, not just quarterly.
Pilot a real text-to-video study today.
Free credits on new accounts, pay-per-response after. No contracts, no annotation ops.