# Text-to-speech human evaluation

> Human evaluation for TTS models via API: pairwise naturalness preferences and MOS ratings from native listeners, at a flat rate per response. Behind the 315K-vote Audio Bench dataset.

Naturalness, prosody and pronunciation are judged by ear, and automatic MOS predictors rank TTS models differently from listeners. Collect pairwise preferences and MOS ratings from native speakers through one API, with the pipeline behind our 315K-vote Audio Bench dataset.

## Published proof

- datapointai/text-to-speech-human-preferences-315k: 315K pairwise votes, 15 models (Open on Hugging Face) at https://huggingface.co/datasets/datapointai/text-to-speech-human-preferences-315k

## What teams collect

### Pairwise naturalness
Two clips of the same text from different models or checkpoints, one vote for the more natural voice. Aggregates into win rates and Elo rankings.

### MOS ratings
A 1 to 5 rating per clip for naturalness or overall quality, with every listener’s score returned so you can compute MOS and its confidence interval yourself.

### Native-listener panels by locale
Target listeners by country and screen for language, so Hindi is judged in India and Spanish in Mexico or Spain. Screened-out listeners are not billed.

### Voice-agent and ad-copy prompts
Test on the text your product reads aloud: support dialogue, numbers and addresses, ad scripts. Rankings change with the prompt set.

## FAQ

### What does human evaluation of a TTS model cost?
You pay a flat rate per response. On platforms that bill participant hours, a 40-second speech comparison comes to about $190 per 1,000 responses; Datapoint is about a quarter of that. Book a call for a quote.

### Should I run MOS or pairwise comparisons?
Pairwise when you are choosing between models or checkpoints: listeners agree more on which clip is better than on an absolute score, so you need fewer votes to separate close systems. MOS when you need an absolute number to report or to track one system over time.

### How do you know listeners heard the clips?
Listeners have to play both clips before they can vote, and playback and response timing are recorded with each vote.

### Which audio formats does the API accept?
MP3, WAV, OGG, M4A and FLAC, uploaded or as public HTTPS URLs. Listeners get a player control for each clip.

### Can automatic MOS predictors replace listeners?
Not for ranking current models. Predicted MOS is useful for catching broken audio, but the top TTS systems differ on prosody and expressiveness, which listeners hear and predictors score poorly.

---

All of this runs on the Datapoint human preference data API: https://trydatapoint.com/human-preference-data-api (docs: https://trydatapoint.com/docs).