// Case study · Voice AI · Text to speech

Text to Speech model evaluation via Human Feedback

10K+human preference evaluationsOn simulated CX and BFSI calls in the Indian context
10+languages coveredIndian English, Hindi, Marathi, Tamil, Telugu, Kannada, Punjabi, Bengali, Odia, Malayalam
300+unknown defects found and treatedEvery defect found feeds the next round, so the test set sharpens as fixes ship
80%less time to identify issues at scale88 voices screened in under 48 hours, 34 cloned voices the same day

01 · The problem

TTS models need human judgement across development stages from real listeners to understand how the performance is perceived in real-life applications.

After model training, two main questions remain that only real listeners can answer.

Problem 01

Model precision

Does the model say every word right?

  • PronunciationNames, places and Hindi English words said the way a native speaker says them
  • NormalisationNumbers, currency, dates and abbreviations read out the right way
  • ProsodyPace, pauses and stress that sound like a person, not a reader
Problem 02

Voice preference

Which voice should speak to your customer?

  • NaturalnessSounds like a real person on a call
  • Suitability / ToneFits the call: firm for collections, warm for support

Global public benchmarks miss two things:

  • Native-speaker perception: they rarely use native listeners, so accent, prosody and pronunciation issues go unnoticed.
  • Use-case fit: they don’t test for the applications a company actually ships, such as customer support, narration or voice agents.

Global English benchmark

  1. #1Lab A
  2. #10Lab B
  3. #44Our client

Judged by US and UK listeners, in English.

Native Hindi listeners · real support lines

  1. #1Our client
  2. #2Lab B
  3. #3Lab A

Our client’s model outputs were consistently rated more natural across all lenses by native Indic speakers.

Global ranks of 90 models from the Artificial Analysis arena, 17 Sep 2026. Native results from twelve listeners on sentences taken from real support calls.

02 · Findings

Findings only humans could have caught

Finding 1 · Industry fit

Collections and support need different voices

Only 2 of 30 voices were best in class on both collections and support. Most of the rest were good at only one.

13001300140014001500150016001600Collections EloSupport Eloworks for both

TakeawayPick a voice per call type, not one voice for every call.

Finding 2 · Whole conversations

A voice that pleases on one line can sound off across a whole call

6 of 9 calls rated bad for a voice in the top half on single lines

Rated alone, one lineJudged as a whole call#1#166 of 9 bad

TakeawayTest voices on full simulated calls, not clips rated alone, as we do at RealLoop.

Finding 3 · Dataset curation

Breakage only shows on a test set built to find it

2k+ test cases across 120 text patterns and 9 industries, curated by humans and AI together

What the cases check · one case can check several

Currency523
Dates512
Numbers318
Mixed language200
Abbreviations169
Times158

TextYour EMI of Rs. 3,450/- is due on 05/09/26.

Right answer… three thousand four hundred fifty rupees is due on fifth September

TakeawayCurate cases by use case and write the right answer before testing. Efficiency and accuracy both need the right human and LLM system.

Finding 4 · Prosody

Voices that run too slow or too fast lose on Elo

40 Elo lower for voices whose pacing runs slower or faster than normal

Normal paceठीक है, WhatsApp पर payment link send कर दी है, कृपया जल्द से जल्द payment कर दीजिएगा, छह august payment की last date है, धन्यवाद।
0:00
Slow paceठीक है, WhatsApp पर payment link send कर दी है, कृपया जल्द से जल्द payment कर दीजिएगा, छह august payment की last date है, धन्यवाद।
0:00
Fast paceठीक है, WhatsApp पर payment link send कर दी है, कृपया जल्द से जल्द payment कर दीजिएगा, छह august payment की last date है, धन्यवाद।
0:00

TakeawayPace affects a voice’s Elo and perceived quality, so human feedback on the pace threshold matters.

03 · Outcome

Learnings and Insights delivered

Voice selection

A shortlist for every language and every kind of call

The best voice changes with the language and with the call. The lab got a ranked shortlist for 7 languages, and for Hindi and English, which voices fit collections and which fit support.

Checkpoint selection

The right checkpoint per language, and a fix list for each

The newer checkpoint won in Hindi, but not everywhere. Outside Hindi, the older checkpoint’s voices led 6 of 10 language ladders. Every defect went back as a fix the team could hear.

Voice evaluation report · shared with the client
HindiIndian EnglishTamilTeluguKannadaBengaliPunjabi

Voice preference by gender · Elo

Women

  1. 1Voice A1613
  2. 2Voice B1586
  3. 3Voice C1562
  4. 4Voice D1444

Men

  1. 1Voice E1558
  2. 2Voice F1540
  3. 3Voice G1532
  4. 4Voice H1526

Best voice by industry · win rate in that call type

BFSI

Voice Awoman83%

Voice Fman83%

Voice Cwoman73%

Customer support

Voice Iman71%

Voice Awoman66%

Voice Hman64%

Conversational

Voice Bwoman89%

Voice Eman75%

Voice Gman66%

Put native speakers on your voices

Send us the lines your agent actually says. We come back with a ladder, the errors that matter, and a retest date.

Public arena ranks from Artificial Analysis, 17 Sep 2026. All other figures from RealLoop’s evaluation program.

© 2026 RealLoop · Bengaluru