Evaluating Motion in Video Generation: 145K Human Preferences and the Limits of VLM Judges
March 31, 2026
In this comparison, 95% of annotators preferred Veo 3 (right) on coherence, while Gemini selected Grok (left).
"A teacher and student spot each other in a bustling airport arrivals hall, rushing together with wide smiles and an enthusiastic embrace."
This pattern is representative of the broader dataset. Across 3,000 VLM-judged comparisons, both Gemini 3 Flash and Qwen 3.5 rank Grok first overall, while human annotators rank Veo first by 7 percentage points.
We collected 145,632 human pairwise preferences on motion-rich video generation across four frontier models (Veo 3 Fast, Grok Imagine, Kling 1.5 Pro, Luma Ray 2), 440 prompts, and 11 content categories. 12,167 annotators evaluated 7,809 comparisons (~19 judgments each). We benchmarked two VLM judges on 3,000 of these same comparisons.
Key findings:
- Veo 3 Fast ranks first (57.7% win rate), leading Grok by 6.9 pp (
p < 0.001) - VLMs rank Grok first (83.2% Gemini, 73.4% Qwen), while humans give Grok 50.8%
- Coherence is the primary driver: humans penalize motion artifacts more heavily than VLMs do
- 10 labels suffice for stable rankings; our ~19 per comparison provide headroom
Study Design
Prompts. 440 prompts across 11 motion-heavy categories (walking, dancing, sports, cooking, animals, etc.), 40 per category. Each prompt combines a unique (subject, action, setting) triple, sampled from curated pools and checked for physical coherence.
Models. Veo 3 Fast, Grok Imagine, Kling 1.5 Pro, Luma Ray 2. Every prompt submitted to every model via Fal.ai — single inference, no cherry-picking. All videos 4–5s at 540–720p.
Comparisons. All pairwise combinations per prompt, evaluated on three dimensions:
- Coherence: temporal consistency, physical plausibility, motion realism
- Aesthetic: composition, lighting, color, production quality
- Prompt adherence: fidelity to the input text
This yields 7,809 unique tasks, each evaluated by ~19 independent annotators via forced-choice comparison.
Data Quality
Response Time
Annotators view two videos side by side (randomized position), with the submit button disabled until both finish playing. Minimum possible time: ~8–10s.
Median response time: 13.3s (3–5s deliberation after watching). The right tail (mean: 15.6s) reflects comparisons requiring repeated viewing.
Inter-Annotator Agreement
| Dimension | Majority Agreement | Strong (80%+) | Moderate (60–80%) | Weak (<60%) |
|---|---|---|---|---|
| Coherence | 67.4% | 450 | 1,393 | 760 |
| Aesthetic | 66.3% | 424 | 1,314 | 865 |
| Prompt Adherence | 65.0% | 303 | 1,376 | 924 |
Overall: 66.2%. For reference, Pick-a-Pic (NeurIPS 2023) reports 68.0% under easier conditions (static images, tie option). Over 35% of our tasks exceed 70% agreement. Coherence has the highest agreement — motion artifacts are more objectively detectable than aesthetic preferences.
Consensus Stability
Even N = 3 labels per comparison produce the correct ranking order (max std < 0.4%). By N = 10, variability is negligible. Our ~19 labels per comparison provide substantial headroom.
Annotator Diversity
12,167 annotators, median 6 labels each. Top 10% contributed 44.6% of labels.
Results reflect a broad population, not a small group of trained raters.
Results
Overall Rankings
| Model | Win Rate | 95% Wilson CI |
|---|---|---|
| Veo 3 Fast | 57.7% | [57.3%, 58.1%] |
| Grok Imagine | 50.8% | [50.4%, 51.2%] |
| Kling 1.5 Pro | 46.1% | [45.8%, 46.5%] |
| Luma Ray 2 | 45.4% | [45.0%, 45.8%] |
Veo 3 Fast ranks first with a 6.9 pp lead over Grok (p < 0.001, 55.4% head-to-head). Grok ranks second, above both Kling and Luma (p < 0.001). Kling and Luma are statistically close (50.8% / 49.2%, p = 0.012).
A Bradley-Terry model fit to all pairwise outcomes produces the same ranking (max deviation: 0.3 pp), confirming internal consistency.
Head-to-Head Matrix
All pairwise matchups are statistically significant:
- Veo 3 Fast wins 55.4% vs Grok, 58.8% vs Kling, 58.9% vs Luma
- Grok Imagine wins 53.6% vs Kling, 54.1% vs Luma
- Kling vs Luma: 50.8% / 49.2% — the narrowest margin in the study
Subcategory Breakdown
Veo ranks first in 9 of 11 categories. Largest margins: animals (62.5%), water (61.4%), cooking (60.2%). Exception: sports/action (50.7%, ns).
Grok ranks first in sports/action, the one category where it outperforms Veo head-to-head (61.6%, p < 10⁻²⁸). Its 64.8% overall sports win rate is the highest single-model rate in any category.
Kling ranks first in stationary content (54.1%, p < 10⁻¹¹), above both Grok and Luma on subtle motion (breathing, posing). Lowest on animals (40.5%) and water (39.6%).
Luma's best category is talking/expressions (50.4%, ns). Worst: dancing (37.5%).
Dimension x Subcategory
Grok's sports advantage holds across all three dimensions. Kling's stationary advantage is significant for coherence and aesthetic but not prompt adherence. Veo's animal lead is uniform across all dimensions.
Hardest Comparisons
The lowest-agreement tasks cluster around Grok vs Kling (152 of 780 bottom-10% tasks) — the matchup with the greatest annotator disagreement and highest human–VLM divergence.
VLM Judge Comparison
We evaluated Gemini 3 Flash and Qwen 3.5 397B on a stratified 3,000-task subset (6,584 judgments). Both support native video input (no lossy frame extraction).
Each VLM saw the same two videos as humans, in randomized order. For coherence and aesthetic, the VLM received no text prompt (matching human conditions). For prompt adherence, the text prompt was included.
Agreement with Humans
| VLM | Overall | High Consensus (≥70%) | Low Consensus (<60%) |
|---|---|---|---|
| Gemini 3 Flash | 62.8% (n=3,373) | 71.0% | 52.1% |
| Qwen 3.5 397B | 59.9% (n=2,798) | 69.0% | 51.2% |
Overall agreement with humans is 60–63%. On ambiguous tasks (<60% human agreement), VLMs are near chance. The two VLMs agree with each other 76% of the time, suggesting shared biases rather than independent error.
VLM Ranking vs. Human Ranking
| Model | Human Win Rate | Gemini Win Rate | Qwen Win Rate |
|---|---|---|---|
| Grok Imagine | 50.8% | 83.2% | 73.4% |
| Veo 3 Fast | 57.7% | 68.5% | 64.1% |
| Kling 1.5 Pro | 46.1% | 26.7% | 37.1% |
| Luma Ray 2 | 45.4% | 21.6% | 25.3% |
Human ranking: Veo > Grok > Kling > Luma. VLM ranking: Grok > Veo > Kling > Luma. This divergence is consistent with public leaderboards that rely on VLM judges.
Dimension-Level Analysis
Two mechanisms contribute to the ranking divergence. First, humans penalize Grok most on coherence:
| Dimension | Grok vs Veo3 (Human) | Significance |
|---|---|---|
| Prompt Adherence | Grok 47.2% | p = 5.2 × 10⁻⁷ |
| Aesthetic | Grok 45.1% | p = 4.7 × 10⁻¹⁸ |
| Coherence | Grok 41.3% | p = 1.4 × 10⁻⁵⁴ |
Grok loses on all three dimensions, but the coherence penalty is largest — 6 pp below prompt adherence.
"A break dancer in a backward cap showcases intricate footwork on a polished grand ballroom floor, the soft glow of glittering chandeliers illuminating the scene."
Second, VLMs inflate Grok across all dimensions — but not uniformly:
| Dimension | Human | Gemini | Δ Gemini | Qwen | Δ Qwen |
|---|---|---|---|---|---|
| Coherence | 46.3% | 74.8% | +28.5 pp | 64.9% | +18.6 pp |
| Prompt Adherence | 54.6% | 83.1% | +28.5 pp | 79.9% | +25.4 pp |
| Aesthetic | 51.4% | 91.5% | +40.0 pp | 78.3% | +26.9 pp |
The aesthetic dimension shows the largest divergence (+40 pp Gemini, +27 pp Qwen). VLMs detect the coherence penalty in direction but underestimate its magnitude. Combined, aesthetic overweighting and coherence underweighting produce the ranking divergence.
A follow-up validation on 253 non-overlapping Grok vs Veo3 prompts (20 fresh responses each) reproduced win rates within 1.9 pp and preserved the dimension ordering — coherence (39.4%) < aesthetic (43.9%) < prompt adherence (45.6%). The coherence penalty is a stable model property, not a sampling artifact.
Implications for VLM-as-Judge
VLMs overweight aesthetic quality (even prompt-blind) and underweight temporal coherence relative to human annotators. For motion-focused evaluation, per-dimension calibration against human judgments is necessary for reliable automated assessment.
Applications
Reward model training. 145,632 pairwise preferences with prompt, model IDs, winner, dimension, and category — suitable for dimension-specific RLHF.
VLM judge calibration. Run additional VLMs on the same 7,809 tasks and compare against human consensus per dimension and category. The full dataset will be published on HuggingFace — follow @datapointai for the release.
Model benchmarking. Generate videos for the same prompts and evaluate through Datapoint for directly comparable results. Sports/action and animals are the most discriminating categories (largest inter-model gaps, highest agreement).
The full study infrastructure is available through Datapoint.
Limitations
Single inference per model. No cherry-picking, but a single poor generation can affect individual prompt scores.
VLM subset. VLM analysis covers 3,000 of 7,809 tasks (~38%). Aggregate and per-dimension results are robust; per-subcategory VLM breakdowns have smaller samples.
Non-expert annotators. 12,167 mobile app users, not video professionals. Good for general preferences; specialized evaluations (e.g., biomechanical accuracy) would benefit from domain experts.
No position logging. The UI randomizes left/right placement but does not log it. The 60-prompt pilot (on HuggingFace) found position bias shifts win rates by <0.05 pp.
Built with Datapoint. Analysis code and the 60-prompt pilot dataset are available at datapointai/VidPrefMotion.
Need this kind of data for your own models? See text-to-video human preference data or the human preference data API.