What Human Evaluation of Generative Models Costs (Video, TTS, Image)
November 9, 2026
A human evaluation budget is two numbers multiplied together: how many responses you need, and what one response costs. This post gives working figures for both, for text-to-video, text-to-speech and image models.
How many responses
For a pairwise study:
responses = prompts x model pairs x votes per pair
Model pairs grow fast. A full round-robin of n models has n(n-1)/2 pairs.
| Study | Prompts | Model pairs | Votes | Responses |
|---|---|---|---|---|
| Checkpoint vs baseline | 500 | 1 | 10 | 5,000 |
| Your model vs 4 competitors | 300 | 4 | 10 | 12,000 |
| 5-model round-robin | 300 | 10 | 10 | 30,000 |
| 15-model leaderboard, round-robin | 300 | 105 | 10 | 315,000 |
Two ways to cut the count without losing the ranking:
- Fewer votes on easy pairs. Lopsided pairs are settled after 5 votes. Spend the rest on close ones.
- Sparse pairings. A new model only needs enough comparisons against existing ones to place it. It does not need the full round-robin.
For MOS, the count is clips x ratings per clip. Pairwise needs fewer votes than MOS to separate two close systems, because listeners agree more on which clip is better than on an absolute score.
What one response costs
There are two pricing models.
Hourly platforms such as Prolific bill participant time plus a fee. For companies that is the hourly reward plus 42.8%: $17.14 an hour at the recommended $12. Cost per response is seconds per response times that rate. See Prolific pricing explained.
Per-response platforms such as Datapoint bill a flat rate per response, the same for every modality. For video and speech it comes to about a quarter of the hourly figure.
| Task | Time per response | Hourly platform, per 1,000 |
|---|---|---|
| Image pair | 15 s | $71 |
| Text or LLM response pair | 30 s | $143 |
| Speech pair, two 10-second clips | 40 s | $190 |
| Video pair, two 10-second clips | 45 s | $214 |
| Video pair with audio, two 20-second clips | 60 s | $286 |
Media that has to be played is where the two models diverge. Longer clips make the hourly bill bigger and leave the per-response bill unchanged.
Worked budgets
Text-to-video, checkpoint vs baseline. 500 prompts, 10 votes: 5,000 responses. About $1,070 on an hourly platform.
Text-to-speech, 5-model round-robin. 300 sentences, 10 pairs, 10 votes: 30,000 responses. About $5,700 hourly.
Video leaderboard, 15 models. 315,000 responses. About $67,000 hourly.
On a per-response platform, expect about a quarter of each figure.
Targeting adds to both. On Prolific, narrow prescreens slow the study. On Datapoint, country or demographic targeting adds a per-response surcharge that is quoted before launch.
The costs that are not on the invoice
- Building the study. On a panel platform you host the task: a page that plays both clips, randomises sides and records the vote. A per-response API hosts it for you.
- Aggregation. Raw exports need turning into win rates, Elo scores and agreement figures.
- Waiting. A study that takes two days blocks a release decision for two days.
A rule of thumb
On an hourly platform, budget 0.5 cents per second of task time. A per-response platform charges the same however long the task is, and for video and speech it costs about a quarter as much.
Email sales@trydatapoint.com for a quote on your own study, or see the modality pages for text-to-video and text-to-speech evaluation.