// Case study · Voice AI · Agent evaluation

Making agent performance measurable

Over and above technical metrics to evaluate whether an agent can hold a call. For a scaled voice AI orchestrator running evaluations for 20+ agents across 8+ languages and 5+ industries.

2.5×
improvement in failure identification
From 30% to 80% of failures caught once human judgment was made part of the evaluation loop
10%
of all call volume reviewed
A steady, representative sample across every deployed agent, scaling as volume grows
20+
agents continuously reviewed
Across 8+ languages and 5+ industries
15+
metrics across the pipeline
One subjective "was this a good call" broken into scores across metrics for STT, LLM, TTS and extraction
01 · The problem

Problem discovery needs human judgment

Discovery breaks at scale

Sampling can't keep up with volume

Listening to real calls and applying judgment is how a team learns what breaks a conversation. That works at small volume. Once the platform is handling serious volume across many deployed agents, there's no bandwidth to sample at a meaningful scale, so issue detection turns sporadic and reactive.

Needs: a steady, representative review that scales with volume, not a heroic listen.

Technical metrics don't tell you a call was good

Automated evals can't hear a robotic voice

Quality was being read through latency, barge-in and similar measures. Those miss what actually matters: how the person on the other end perceives the call, or how the client would judge it if they sampled one.

Needs: the subjective sense of a good call turned into metrics that can each be scored.

Not just a hiring problem

Why it is tough to solve internally

The team tried interns, then a small internal reviewer group. Beyond the operational overhead, the job is not as simple as hiring people. It takes real investment to break the task into reviewable subtasks, build the tooling to capture judgment efficiently, and train and monitor reviewers to keep quality high.

Needs: rubric design, workflow tooling and reviewer calibration as one system.

The RealLoop agent evaluation loop A four-stage cycle: evaluation design, failure diagnosis, expert training data, monitor reliability, then back to design as new failure modes surface. 01 02 03 04 EVALUATION DESIGN FAILURE DIAGNOSIS TRAINING DATA MONITOR RELIABILITY
02 · The method

The loop

Design the metrics, diagnose where the agent fails, build targeted data to fix it, monitor reliability as things change

01Evaluation design
Define success
Success criteria, rubrics and scoring for the agent's task
15+ metrics · 4 pipeline stages
02Failure diagnosis
Find where, why and how it fails
The right system per metric, run on representative volume
tech + human, by metric
03Training data
Data built backwards from failures
Humans produce targeted data for the top failure modes
enable fine-tuning of the model/prompt/workflow
04Monitor reliability
Re-evaluate as things change
Always-on tracking as models, prompts, workflows and rules move
IRR + ground truth
03 · What we measure

One "good call", broken into 15+ metrics

We turned a single subjective sense of quality into metrics that can each be scored on their own. Each metric gets the evaluation it actually needs: a machine, a human, or both.

Human-led Machine Hybrid

Overall

whole-call quality & the extracted outcome
Call naturalnessHuman-led

Does the whole call sound like a real person the customer would trust?

LatencyMachine

Dead air and slow responses that make the agent feel laggy or cut people off.

DispositionHybrid

Does the extracted outcome, like intent to pay, match what really happened?

Speech to text

STT · did the agent hear it right?
TranscriptionHybrid

Overall word accuracy: did the agent hear what was actually said?

Entity captureHybrid

Critical fields (order ID, phone, amount) captured exactly right.

Numbers & codesHybrid

Digits, dates and alphanumeric codes, where one wrong token breaks the call.

Proper nounsHuman-led

Names, places and brands the model most often mishears.

Code-switchHybrid

Mixed-language speech transcribed cleanly, not garbled into nonsense.

Reasoning

LLM · did it understand and act right?
Language adherenceHybrid

Does the agent stay in the language the caller is actually speaking?

Goal completionHybrid

Did the call actually do what the user rang in to get done?

Knowledge retentionHybrid

Does the agent hold on to details the caller gave earlier?

Stuck in loopHybrid

Repeating itself or looping without ever moving the call forward.

Tool-name leakMachine

Internal tool or function names spoken out loud to the caller.

Text to speech

TTS · did it sound right?
Barge-inMachine

Does the agent stop talking the moment the caller interrupts?

PronunciationHuman-led

Wrong accents and mispronounced names, places and terms across languages.

ToneHuman-led

Right words, wrong emotion: flat, robotic, or mismatched to the moment.

04 · Findings

What the breakdown revealed

STT · speech to text
Values the call depends on, captured wrong

The agent records the wrong value for a field the outcome hinges on

Input capture errors: wrong order ID, wrong phone numbera field the call depends on
Order BR-4021 captured as BR-4201; "double five" logged as "5"example
Transcription breaks on proper nouns: names, places, brandsderails comprehension of the call
"Andheri" heard as "Andhary"; "Bengaluru" as "Bengalow"example
Code-switched speech breaks in ways single-language audio doesn'tacross 8+ languages
LLM · reasoning
Failures the LLM judge can't flag

Agent misses the goal, drifts language, loops, or leaks its own plumbing to the caller.

Goal completion: the call ends without doing what the user askedthe outcome that matters
Caller asks to reschedule delivery; agent confirms the address, then hangs up without rebookingexample
Language adherence: drifts out of the requested language mid-callhybrid, calibrated to humans
Caller speaks Hindi; agent answers in English from turn three onwardexample
Stuck in a loop: repeats itself and can't recoverLLM judge, human-calibrated
Tool-name leak: internal function names spoken to the callermachine-detectable
Agent says "let me run check_order_status" out loudexample
Knowledge retention: loses a detail the caller gave earliercontext the agent should hold
TTS · text to speech
Right words, wrong delivery

Tone and pronunciation live outside the transcript entirely. Only a listener catches them, and they decide whether the agent sounds human.

Tone failures: correct words, flat or wrong emotionthe caller sounds robotic
"Good news, your refund is approved" read in a flat, sad toneexample
Pronunciation: wrong accents, mispronounced proper nounshuman-led, across languages
PAN spoken as "pain", KYC as "kyz"example
Extraction
The judge scoring the call can be wrong

Disposition, often a call's most important output, comes from an LLM judge that had never been measured against a human. We compute its accuracy and plug humans in where it's most likely to be wrong.

Disposition accuracy: e.g. intent-to-pay on collections callsLLM judge, benchmarked to humans
Borrower says "I'll try to arrange something"; judge labels it "will pay"example
On one fintech deployment, the judge mislabeled intent30% of the time
Accuracy on multi-intent calls vs single-intent45% against 85%
Humans resolve the high-risk cases and feed the judge backblended accuracy held
05 · Into production & beyond

Measurement that keeps up with the agent

A new failure mode isn't a surprise. It's the next thing we build a measurement for.

The engagement becomes a business-as-usual pipeline that reviews ~10% of calls across agent types, languages and industries. As fresh problems surface we design the way to track each one at scale, and build a human or an automated evaluation depending on what the metric actually needs.

Always-on tracking

A representative sample scored continuously, so regressions show up as models, prompts and rules change.

~10% of volume

Reliability, measured

Agent reliability read through inter-rater agreement and comparison against a ground-truth set.

IRR + ground truth

A playbook that ports

The rubric design, workflow tooling and reviewer training deploy across new use-cases with high confidence.

reusable by design

Put human judgment on your agents

Send us the calls your agent actually handles. We come back with a metric breakdown, the failures that matter, and a monitoring pipeline that scales.

Contact us