# Evaluating Motion in Video Generation: 145K Human Preferences and the Limits of VLM Judges

> A large-scale human preference study comparing Veo 3, Grok Imagine, Kling 1.5 Pro, and Luma Ray 2 across 440 prompts and 11 content categories, with VLM judge benchmarking revealing systematic biases in automated evaluation.

In this comparison, 95% of annotators preferred Veo 3 (right) on coherence, while Gemini selected Grok (left).

<VideoPair
  videoA="https://v3b.fal.media/files/b/0a9013f5/HGNnvNvG4IUgBQDSfzuar_6ZtRvZTh.mp4"
  videoB="https://v3b.fal.media/files/b/0a9013f9/FQuXk8NQYlNi2YiK_Hrim_6fc3bc29255b4f0b9a3becfebf55ea4e.mp4"
  labelA="Grok Imagine"
  labelB="Veo 3 Fast"
  prompt="A teacher and student spot each other in a bustling airport arrivals hall, rushing together with wide smiles and an enthusiastic embrace."
  humanWinner="b"
  vlmWinner="a"
  humanAgreement="95%"
  dimension="coherence"
  subcategory="multi_person"
  eager
/>

This pattern is representative of the broader dataset. Across 3,000 VLM-judged comparisons, both Gemini 3 Flash and Qwen 3.5 rank Grok first overall, while human annotators rank Veo first by 7 percentage points.

<VideoGallery
  buttonLabel="Browse more examples"
  items={[
    {
      videoA: "https://v3b.fal.media/files/b/0a8f8a07/_qWFl_duUytkuS71gay8b_L0yTn9LP.mp4",
      videoB: "https://v3b.fal.media/files/b/0a8f8a06/vq8LpFvCxQGNbqEmNN8kS_0d35583aa01942c6896c252605cd5e0a.mp4",
      labelA: "Grok Imagine",
      labelB: "Veo 3 Fast",
      prompt: "A construction worker in a bright yellow hard hat and reflective vest strides confidently through a quiet cemetery at twilight.",
      humanWinner: "b",
      vlmWinner: "a",
      humanAgreement: "94%",
      dimension: "coherence",
      subcategory: "walking"
    },
    {
      videoA: "https://v3b.fal.media/files/b/0a901256/otWYYix1hAe-DsC-9fowH_q5x61kzp.mp4",
      videoB: "https://v3b.fal.media/files/b/0a901259/RVjyabcxN7QMP8r5qI36y_f81fda42e98a4d11922f5e886bce2ac0.mp4",
      labelA: "Grok Imagine",
      labelB: "Veo 3 Fast",
      prompt: "Two old friends reunite in a sunlit classroom, wrapping each other in a warm, tearful embrace beside rows of empty desks.",
      humanWinner: "b",
      vlmWinner: "a",
      humanAgreement: "94%",
      dimension: "coherence",
      subcategory: "multi_person"
    },
    {
      videoA: "https://v3b.fal.media/files/b/0a901334/ZA3C2_NvJSc2Jkt6rTMqY_output.mp4",
      videoB: "https://v3b.fal.media/files/b/0a901324/zme5IDCwEGumYcwBG7a3s_f2e6307af0d34f3882569be52f3637c9.mp4",
      labelA: "Kling 1.5 Pro",
      labelB: "Veo 3 Fast",
      prompt: "A synchronized swimmer in a sparkly suit spins gracefully upright, her sequins catching the light in a crystal-clear Olympic pool.",
      humanWinner: "b",
      vlmWinner: "a",
      humanAgreement: "100%",
      dimension: "aesthetic",
      subcategory: "water"
    },
    {
      videoA: "https://v3b.fal.media/files/b/0a901295/V4nFsEj0ZbnQchshRvYi6_output.mp4",
      videoB: "https://v3b.fal.media/files/b/0a901286/TgXLBXUvoq9VjuS-Y368E_59e599f81f6643e9bc1a141e3125beb4.mp4",
      labelA: "Kling 1.5 Pro",
      labelB: "Veo 3 Fast",
      prompt: "A skilled chocolatier spreads and folds glossy dark chocolate across a cool marble slab, steam rising in a pristine professional workshop.",
      humanWinner: "b",
      vlmWinner: "a",
      humanAgreement: "100%",
      dimension: "aesthetic",
      subcategory: "cooking"
    },
    {
      videoA: "https://v3b.fal.media/files/b/0a901208/lOqH9y3_hXA5o5I3jzYyg_65jHi8Av.mp4",
      videoB: "https://v3b.fal.media/files/b/0a901232/YJJ8kApH-KWd0RkOe9LoU_output.mp4",
      labelA: "Grok Imagine",
      labelB: "Luma Ray 2",
      prompt: "A calligrapher sits cross-legged outdoors, brush in hand, reading intently from an ancient text surrounded by lush forest greenery.",
      humanWinner: "b",
      vlmWinner: "a",
      humanAgreement: "100%",
      dimension: "coherence",
      subcategory: "stationary"
    },
    {
      videoA: "https://v3b.fal.media/files/b/0a901314/wv7dHcOnFkBBtidH2xswT_output.mp4",
      videoB: "https://v3b.fal.media/files/b/0a90131c/FEcyvUwb9XMYoV9LPzVJG_output.mp4",
      labelA: "Kling 1.5 Pro",
      labelB: "Luma Ray 2",
      prompt: "A synchronized swimmer in a sparkly suit splashes joyfully in a sunlit ocean cove, her sequins scattering light across the shimmering shallow water.",
      humanWinner: "b",
      vlmWinner: "a",
      humanAgreement: "100%",
      dimension: "coherence",
      subcategory: "water"
    }
  ]}
/>

We collected **145,632 human pairwise preferences** on motion-rich video generation across four frontier models (Veo 3 Fast, Grok Imagine, Kling 1.5 Pro, Luma Ray 2), 440 prompts, and 11 content categories. 12,167 annotators evaluated 7,809 comparisons (~19 judgments each). We benchmarked two VLM judges on 3,000 of these same comparisons.

**Key findings:**
- **Veo 3 Fast ranks first** (57.7% win rate), leading Grok by 6.9 pp (`p < 0.001`)
- **VLMs rank Grok first** (83.2% Gemini, 73.4% Qwen), while humans give Grok 50.8%
- **Coherence is the primary driver**: humans penalize motion artifacts more heavily than VLMs do
- **10 labels suffice** for stable rankings; our ~19 per comparison provide headroom

## Study Design

**Prompts.** 440 prompts across 11 motion-heavy categories (walking, dancing, sports, cooking, animals, etc.), 40 per category. Each prompt combines a unique (subject, action, setting) triple, sampled from curated pools and checked for physical coherence.

**Models.** Veo 3 Fast, Grok Imagine, Kling 1.5 Pro, Luma Ray 2. Every prompt submitted to every model via Fal.ai — single inference, no cherry-picking. All videos 4–5s at 540–720p.

**Comparisons.** All pairwise combinations per prompt, evaluated on three dimensions:
- **Coherence**: temporal consistency, physical plausibility, motion realism
- **Aesthetic**: composition, lighting, color, production quality
- **Prompt adherence**: fidelity to the input text

This yields 7,809 unique tasks, each evaluated by ~19 independent annotators via forced-choice comparison.

## Data Quality

### Response Time

Annotators view two videos side by side (randomized position), with the submit button disabled until both finish playing. Minimum possible time: ~8–10s.

![Response time distribution](/images/blog/vidprefmotion/response_time.png)

Median response time: **13.3s** (3–5s deliberation after watching). The right tail (mean: 15.6s) reflects comparisons requiring repeated viewing.

### Inter-Annotator Agreement

![Agreement distribution](/images/blog/vidprefmotion/agreement_dist.png)

| Dimension | Majority Agreement | Strong (80%+) | Moderate (60–80%) | Weak (`<`60%) |
|---|---|---|---|---|
| Coherence | 67.4% | 450 | 1,393 | 760 |
| Aesthetic | 66.3% | 424 | 1,314 | 865 |
| Prompt Adherence | 65.0% | 303 | 1,376 | 924 |

**Overall: 66.2%.** For reference, Pick-a-Pic (NeurIPS 2023) reports 68.0% under easier conditions (static images, tie option). Over 35% of our tasks exceed 70% agreement. Coherence has the highest agreement — motion artifacts are more objectively detectable than aesthetic preferences.

### Consensus Stability

![Convergence](/images/blog/vidprefmotion/convergence.png)

Even **N = 3** labels per comparison produce the correct ranking order (max std `<` 0.4%). By N = 10, variability is negligible. Our ~19 labels per comparison provide substantial headroom.

### Annotator Diversity

12,167 annotators, median 6 labels each. Top 10% contributed 44.6% of labels.

![Annotator distribution](/images/blog/vidprefmotion/annotator_dist.png)

Results reflect a broad population, not a small group of trained raters.

## Results

### Overall Rankings

![Overall rankings](/images/blog/vidprefmotion/overall_rankings.png)

| Model | Win Rate | 95% Wilson CI |
|---|---|---|
| Veo 3 Fast | 57.7% | [57.3%, 58.1%] |
| Grok Imagine | 50.8% | [50.4%, 51.2%] |
| Kling 1.5 Pro | 46.1% | [45.8%, 46.5%] |
| Luma Ray 2 | 45.4% | [45.0%, 45.8%] |

**Veo 3 Fast ranks first** with a 6.9 pp lead over Grok (`p < 0.001`, 55.4% head-to-head). Grok ranks second, above both Kling and Luma (`p < 0.001`). **Kling and Luma are statistically close** (50.8% / 49.2%, p = 0.012).

A Bradley-Terry model fit to all pairwise outcomes produces the same ranking (max deviation: 0.3 pp), confirming internal consistency.

![Bradley-Terry Scores](/images/blog/vidprefmotion/bradley_terry.png)

### Head-to-Head Matrix

![Head-to-head](/images/blog/vidprefmotion/h2h_heatmap.png)

All pairwise matchups are statistically significant:
- **Veo 3 Fast** wins 55.4% vs Grok, 58.8% vs Kling, 58.9% vs Luma
- **Grok Imagine** wins 53.6% vs Kling, 54.1% vs Luma
- **Kling vs Luma**: 50.8% / 49.2% — the narrowest margin in the study

### Subcategory Breakdown

![Model x Subcategory](/images/blog/vidprefmotion/model_x_subcat.png)

**Veo ranks first in 9 of 11 categories.** Largest margins: animals (62.5%), water (61.4%), cooking (60.2%). Exception: sports/action (50.7%, ns).

**Grok ranks first in sports/action**, the one category where it outperforms Veo head-to-head (61.6%, `p < 10⁻²⁸`). Its 64.8% overall sports win rate is the highest single-model rate in any category.

**Kling ranks first in stationary content** (54.1%, `p < 10⁻¹¹`), above both Grok and Luma on subtle motion (breathing, posing). Lowest on animals (40.5%) and water (39.6%).

**Luma's best category is talking/expressions** (50.4%, ns). Worst: dancing (37.5%).

### Dimension x Subcategory

![Dimension x Subcategory breakdown](/images/blog/vidprefmotion/dim_x_subcat.png)

Grok's sports advantage holds across all three dimensions. Kling's stationary advantage is significant for coherence and aesthetic but not prompt adherence. Veo's animal lead is uniform across all dimensions.

### Hardest Comparisons

The lowest-agreement tasks cluster around **Grok vs Kling** (152 of 780 bottom-10% tasks) — the matchup with the greatest annotator disagreement and highest human–VLM divergence.

## VLM Judge Comparison

We evaluated **Gemini 3 Flash** and **Qwen 3.5 397B** on a stratified 3,000-task subset (6,584 judgments). Both support native video input (no lossy frame extraction).

Each VLM saw the same two videos as humans, in randomized order. For coherence and aesthetic, the VLM received no text prompt (matching human conditions). For prompt adherence, the text prompt was included.

### Agreement with Humans

| VLM | Overall | High Consensus (≥70%) | Low Consensus (`<`60%) |
|---|---|---|---|
| Gemini 3 Flash | 62.8% (n=3,373) | 71.0% | 52.1% |
| Qwen 3.5 397B | 59.9% (n=2,798) | 69.0% | 51.2% |

Overall agreement with humans is 60–63%. On ambiguous tasks (`<`60% human agreement), VLMs are near chance. The two VLMs agree with *each other* **76%** of the time, suggesting shared biases rather than independent error.

![VLM Agreement by Dimension](/images/blog/vidprefmotion/vlm_agreement_by_dim.png)

### VLM Ranking vs. Human Ranking

| Model | Human Win Rate | Gemini Win Rate | Qwen Win Rate |
|---|---|---|---|
| Grok Imagine | 50.8% | 83.2% | 73.4% |
| Veo 3 Fast | 57.7% | 68.5% | 64.1% |
| Kling 1.5 Pro | 46.1% | 26.7% | 37.1% |
| Luma Ray 2 | 45.4% | 21.6% | 25.3% |

Human ranking: Veo > Grok > Kling > Luma. VLM ranking: Grok > Veo > Kling > Luma. This divergence is consistent with public leaderboards that rely on VLM judges.

![VLM Win Rate Comparison](/images/blog/vidprefmotion/vlm_win_rate_comparison.png)

### Dimension-Level Analysis

Two mechanisms contribute to the ranking divergence. First, humans penalize Grok most on coherence:

| Dimension | Grok vs Veo3 (Human) | Significance |
|---|---|---|
| Prompt Adherence | Grok 47.2% | p = 5.2 × 10⁻⁷ |
| Aesthetic | Grok 45.1% | p = 4.7 × 10⁻¹⁸ |
| Coherence | Grok 41.3% | p = 1.4 × 10⁻⁵⁴ |

Grok loses on all three dimensions, but the coherence penalty is largest — 6 pp below prompt adherence.

<VideoPair
  videoA="https://v3b.fal.media/files/b/0a8f8a22/UfUzAlk8FvaPYznxpxapK_FyL040Zk.mp4"
  videoB="https://v3b.fal.media/files/b/0a8f8a22/0OrVG47Gf1XxaJhmLdAHp_aac370096bf84d9e92fba19e5dc7f42c.mp4"
  labelA="Grok Imagine"
  labelB="Veo 3 Fast"
  prompt="A break dancer in a backward cap showcases intricate footwork on a polished grand ballroom floor, the soft glow of glittering chandeliers illuminating the scene."
  humanWinner="b"
  vlmWinner="a"
  humanAgreement="90%"
  dimension="coherence"
  subcategory="dancing"
/>

Second, VLMs inflate Grok across all dimensions — but not uniformly:

| Dimension | Human | Gemini | Δ Gemini | Qwen | Δ Qwen |
|---|---|---|---|---|---|
| Coherence | 46.3% | 74.8% | +28.5 pp | 64.9% | +18.6 pp |
| Prompt Adherence | 54.6% | 83.1% | +28.5 pp | 79.9% | +25.4 pp |
| Aesthetic | 51.4% | 91.5% | +40.0 pp | 78.3% | +26.9 pp |

The aesthetic dimension shows the largest divergence (+40 pp Gemini, +27 pp Qwen). VLMs detect the coherence penalty in direction but underestimate its magnitude. Combined, aesthetic overweighting and coherence underweighting produce the ranking divergence.

![Grok's Coherence Gap](/images/blog/vidprefmotion/vlm_grok_coherence_gap.png)

A follow-up validation on 253 non-overlapping Grok vs Veo3 prompts (20 fresh responses each) reproduced win rates within 1.9 pp and preserved the dimension ordering — coherence (39.4%) `<` aesthetic (43.9%) `<` prompt adherence (45.6%). The coherence penalty is a stable model property, not a sampling artifact.

### Implications for VLM-as-Judge

VLMs overweight aesthetic quality (even prompt-blind) and underweight temporal coherence relative to human annotators. For motion-focused evaluation, per-dimension calibration against human judgments is necessary for reliable automated assessment.

## Applications

**Reward model training.** 145,632 pairwise preferences with prompt, model IDs, winner, dimension, and category — suitable for dimension-specific RLHF.

**VLM judge calibration.** Run additional VLMs on the same 7,809 tasks and compare against human consensus per dimension and category. The full dataset will be published on HuggingFace — follow [@datapointai](https://huggingface.co/datapointai) for the release.

**Model benchmarking.** Generate videos for the same prompts and evaluate through [Datapoint](https://trydatapoint.com) for directly comparable results. Sports/action and animals are the most discriminating categories (largest inter-model gaps, highest agreement).

The full study infrastructure is available through [Datapoint](https://trydatapoint.com).

## Limitations

**Single inference per model.** No cherry-picking, but a single poor generation can affect individual prompt scores.

**VLM subset.** VLM analysis covers 3,000 of 7,809 tasks (~38%). Aggregate and per-dimension results are robust; per-subcategory VLM breakdowns have smaller samples.

**Non-expert annotators.** 12,167 mobile app users, not video professionals. Good for general preferences; specialized evaluations (e.g., biomechanical accuracy) would benefit from domain experts.

**No position logging.** The UI randomizes left/right placement but does not log it. The 60-prompt pilot ([on HuggingFace](https://huggingface.co/datasets/datapointai/VidPrefMotion)) found position bias shifts win rates by `<`0.05 pp.

---

*Built with [Datapoint](https://trydatapoint.com). Analysis code and the 60-prompt pilot dataset are available at [datapointai/VidPrefMotion](https://huggingface.co/datasets/datapointai/VidPrefMotion).*

*Need this kind of data for your own models? See [text-to-video human preference data](/preference-data/text-to-video) or the [human preference data API](/human-preference-data-api).*
