Case study · Voice AI · Agent evaluation

Making agent performance measurable

Over and above technical metrics, to evaluate whether an agent can hold a call. For a scaled voice AI orchestrator running evaluations for 20+ agents across 8+ languages and 5+ industries.

2.5×improvement in failure identificationFrom 30% to 80% of failures caught once human judgment joined the loop
10%of all call volume reviewedA steady, representative sample across every deployed agent
20+agents continuously reviewedAcross 8+ languages and 5+ industries
15+metrics across the pipelineOne subjective “was this a good call” broken into scores

01 · The problem

Problem discovery needs human judgment

Sampling can’t keep up with volume

Once the platform handles serious volume across many deployed agents, there’s no bandwidth to sample at a meaningful scale, so issue detection turns sporadic and reactive.

Automated evals can’t hear a robotic voice

Latency, barge-in and similar measures miss what actually matters: how the person on the other end perceives the call.

Not just a hiring problem

It takes real investment to break the task into reviewable subtasks, build the tooling to capture judgment, and train and monitor reviewers to keep quality high.

02 · The method

The loop

01

Evaluation design

Success criteria, rubrics and scoring for the agent’s task.

15+ metrics · 4 pipeline stages
02

Failure diagnosis

The right system per metric, run on representative volume.

tech + human, by metric
03

Training data

Humans produce targeted data for the top failure modes.

fine-tuning inputs
04

Monitor reliability

Always-on tracking as models, prompts, workflows and rules move.

IRR + ground truth

Back to design as new failure modes surface

03 · What we measure

One “good call”, broken into 15+ metrics

Each metric gets the evaluation it actually needs: a machine, a human, or both.

Human-ledMachineHybrid

Overall

Whole-call quality and the extracted outcome

Call naturalnessLatencyDisposition

Speech to text

Did the agent hear it right?

TranscriptionEntity captureNumbers and codesProper nounsCode-switch

Reasoning

Did it understand and act right?

Language adherenceGoal completionKnowledge retentionStuck in loopTool-name leak

Text to speech

Did it sound right?

Barge-inPronunciationTone

04 · Findings

What the breakdown revealed

Order BR-4021 captured as BR-4201
Speech to text: a field the call depends on, captured wrong
STT
Caller speaks Hindi; agent answers in English from turn three onward
Reasoning: drifts out of the requested language mid-call
LLM
Agent says “let me run check_order_status” out loud
Reasoning: internal tool names spoken to the caller
LLM
“Good news, your refund is approved” read in a flat, sad tone
Text to speech: right words, wrong emotion
TTS
Borrower says “I’ll try to arrange something”; judge labels it “will pay”
Extraction: the LLM judge scoring the call can be wrong
Judge
30%of intent labels the LLM judge got wrong on one fintech deployment, before humans were plugged in where it is most likely to be wrong.

Put human judgment on your agents

Send us the calls your agent actually handles. We come back with a metric breakdown, the failures that matter, and a monitoring pipeline that scales.

Figures from RealLoop’s agent evaluation program. Fintech disposition figures from the linked collections engagement.

© 2026 RealLoop · Bengaluru