01 · The problem
TTS models need human judgement across development stages from real listeners to understand how the performance is perceived in real-life applications.
After model training, two main questions remain that only real listeners can answer.
Model precision
Does the model say every word right?
- PronunciationNames, places and Hindi English words said the way a native speaker says them
- NormalisationNumbers, currency, dates and abbreviations read out the right way
- ProsodyPace, pauses and stress that sound like a person, not a reader
Voice preference
Which voice should speak to your customer?
- NaturalnessSounds like a real person on a call
- Suitability / ToneFits the call: firm for collections, warm for support
Global public benchmarks miss two things:
- Native-speaker perception: they rarely use native listeners, so accent, prosody and pronunciation issues go unnoticed.
- Use-case fit: they don’t test for the applications a company actually ships, such as customer support, narration or voice agents.
Global English benchmark
- #1Lab A
- #10Lab B
- #44Our client
Judged by US and UK listeners, in English.
Native Hindi listeners · real support lines
- #1Our client
- #2Lab B
- #3Lab A
Our client’s model outputs were consistently rated more natural across all lenses by native Indic speakers.
Global ranks of 90 models from the Artificial Analysis arena, 17 Sep 2026. Native results from twelve listeners on sentences taken from real support calls.
02 · Findings
Findings only humans could have caught
Finding 1 · Industry fit
Collections and support need different voices
Only 2 of 30 voices were best in class on both collections and support. Most of the rest were good at only one.
TakeawayPick a voice per call type, not one voice for every call.
Finding 2 · Whole conversations
A voice that pleases on one line can sound off across a whole call
6 of 9 calls rated bad for a voice in the top half on single lines
TakeawayTest voices on full simulated calls, not clips rated alone, as we do at RealLoop.
Finding 3 · Dataset curation
Breakage only shows on a test set built to find it
2k+ test cases across 120 text patterns and 9 industries, curated by humans and AI together
What the cases check · one case can check several
TextYour EMI of Rs. 3,450/- is due on 05/09/26.
Right answer… three thousand four hundred fifty rupees is due on fifth September
TakeawayCurate cases by use case and write the right answer before testing. Efficiency and accuracy both need the right human and LLM system.
Finding 4 · Prosody
Voices that run too slow or too fast lose on Elo
40 Elo lower for voices whose pacing runs slower or faster than normal
TakeawayPace affects a voice’s Elo and perceived quality, so human feedback on the pace threshold matters.
03 · Outcome
Learnings and Insights delivered
Voice selection
A shortlist for every language and every kind of call
The best voice changes with the language and with the call. The lab got a ranked shortlist for 7 languages, and for Hindi and English, which voices fit collections and which fit support.
Checkpoint selection
The right checkpoint per language, and a fix list for each
The newer checkpoint won in Hindi, but not everywhere. Outside Hindi, the older checkpoint’s voices led 6 of 10 language ladders. Every defect went back as a fix the team could hear.
Voice preference by gender · Elo
Women
- 1Voice A1613
- 2Voice B1586
- 3Voice C1562
- 4Voice D1444
Men
- 1Voice E1558
- 2Voice F1540
- 3Voice G1532
- 4Voice H1526
Best voice by industry · win rate in that call type
BFSI
Voice Awoman83%
Voice Fman83%
Voice Cwoman73%
Customer support
Voice Iman71%
Voice Awoman66%
Voice Hman64%
Conversational
Voice Bwoman89%
Voice Eman75%
Voice Gman66%
Put native speakers on your voices
Send us the lines your agent actually says. We come back with a ladder, the errors that matter, and a retest date.
Public arena ranks from Artificial Analysis, 17 Sep 2026. All other figures from RealLoop’s evaluation program.