# What Human Evaluation of Generative Models Costs (Video, TTS, Image)

> How to budget a human evaluation for text-to-video, text-to-speech and image models: how many responses you need, and what they cost per 1,000 on hourly and per-response platforms.

A human evaluation budget is two numbers multiplied together: how many responses you need, and what one response costs. This post gives working figures for both, for text-to-video, text-to-speech and image models.

## How many responses

For a pairwise study:

```text
responses = prompts x model pairs x votes per pair
```

Model pairs grow fast. A full round-robin of `n` models has `n(n-1)/2` pairs.

| Study                             | Prompts | Model pairs | Votes | Responses |
| --------------------------------- | ------- | ----------- | ----- | --------- |
| Checkpoint vs baseline            | 500     | 1           | 10    | 5,000     |
| Your model vs 4 competitors       | 300     | 4           | 10    | 12,000    |
| 5-model round-robin               | 300     | 10          | 10    | 30,000    |
| 15-model leaderboard, round-robin | 300     | 105         | 10    | 315,000   |

Two ways to cut the count without losing the ranking:

- **Fewer votes on easy pairs.** Lopsided pairs are settled after 5 votes. Spend the rest on close ones.
- **Sparse pairings.** A new model only needs enough comparisons against existing ones to place it. It does not need the full round-robin.

For MOS, the count is `clips x ratings per clip`. Pairwise needs fewer votes than MOS to separate two close systems, because listeners agree more on which clip is better than on an absolute score.

## What one response costs

There are two pricing models.

**Hourly platforms** such as Prolific bill participant time plus a fee. For companies that is the hourly reward plus 42.8%: $17.14 an hour at the recommended $12. Cost per response is seconds per response times that rate. See [Prolific pricing explained](/blog/prolific-pricing-explained).

**Per-response platforms** such as Datapoint bill a flat rate per response, the same for every modality. For video and speech it comes to about a quarter of the hourly figure.

| Task                                       | Time per response | Hourly platform, per 1,000 |
| ------------------------------------------ | ----------------- | -------------------------- |
| Image pair                                 | 15 s              | $71                        |
| Text or LLM response pair                  | 30 s              | $143                       |
| Speech pair, two 10-second clips           | 40 s              | $190                       |
| Video pair, two 10-second clips            | 45 s              | $214                       |
| Video pair with audio, two 20-second clips | 60 s              | $286                       |

Media that has to be played is where the two models diverge. Longer clips make the hourly bill bigger and leave the per-response bill unchanged.

## Worked budgets

**Text-to-video, checkpoint vs baseline.** 500 prompts, 10 votes: 5,000 responses. About $1,070 on an hourly platform.

**Text-to-speech, 5-model round-robin.** 300 sentences, 10 pairs, 10 votes: 30,000 responses. About $5,700 hourly.

**Video leaderboard, 15 models.** 315,000 responses. About $67,000 hourly.

On a per-response platform, expect about a quarter of each figure.

Targeting adds to both. On Prolific, narrow prescreens slow the study. On Datapoint, country or demographic targeting adds a per-response surcharge that is quoted before launch.

## The costs that are not on the invoice

- **Building the study.** On a panel platform you host the task: a page that plays both clips, randomises sides and records the vote. A per-response API hosts it for you.
- **Aggregation.** Raw exports need turning into win rates, Elo scores and agreement figures.
- **Waiting.** A study that takes two days blocks a release decision for two days.

## A rule of thumb

On an hourly platform, budget 0.5 cents per second of task time. A per-response platform charges the same however long the task is, and for video and speech it costs about a quarter as much.

Email sales@trydatapoint.com for a quote on your own study, or see the modality pages for [text-to-video](/preference-data/text-to-video) and [text-to-speech](/preference-data/text-to-speech) evaluation.