Grounded voice insight: turning recorded speech into claims you can check

In one picture

Grounded voice insight is a way to get judgements out of recorded conversations that a reader can check. A speech-to-text model turns the audio into a transcript, which becomes the ground truth: the one record everything else is checked against. Whatever can be counted is measured from it without a model. Whatever needs judgement goes to an LLM judge (Zheng et al., 2023), which must quote the transcript for every claim. A verifier then checks each quote before anything reaches the reader.

Why it matters

LLMs summarise and score conversations fluently, and they also invent plausible quotes. In call review, interviews or clinical notes, an invented quote is a false statement about a real person. A plain "summarise this call" prompt gives the reader no way to tell a grounded claim from a made-up one.

• Which parts of this report are facts, and which are opinions?
• Where in the recording does each claim come from?
• How much of what the model said survived checking?

The building blocks

Part	Holds / does	Everyday example
Speech-to-text	Turns audio into words with timings, speaker labels and a confidence score per word	A court stenographer who also notes the time of every line
Transcript	The ground truth every later step reads and is checked against	The minutes of a meeting
Metrics	Counts from the word timings: talk share, silences, interruptions. No model involved	A stopwatch, not a critic
Rubric	The list of things to judge, written as data the prompt and the output schema both use	A marking scheme
LLM judge	Reads the transcript in its original language and returns claims, each with a verbatim quote, a time and a speaker	A reviewer who must cite the page for every remark
Verifier	Checks each quote: is the text there, near that time, from that speaker? Drops and counts failures	A fact-checker at a newspaper
Report	Shows measured facts and checked claims separately, with how many claims were dropped	An article with footnotes

How it works

1. Transcribe. Speech-to-text returns words, timings and speakers. Don't translate first: the quotes must stay in the language that was spoken.
2. Measure. Compute the countable facts from the timings, so they are the same on every run.
3. Prompt. Give the model the rubric, strict rules ("quote verbatim, copy the times, abstain if you can't judge") and the timestamped transcript.
4. Infer. The model returns structured output in which a quote is a required field of every claim.
5. Verify. For each claim, find the quote in the transcript, check it sits near the claimed time, and check the speaker. Keep it, or drop it and record why.
6. Report. Show the measured facts, the kept claims with their quotes, and the drop count. If the model made claims and none survived, withhold the score.

Patterns & variations

Pattern	What changes	When to use it
Stereo recording	Each speaker is on their own channel, so no diarization guess	When the phone or meeting system records the sides separately
Self-hosted speech-to-text (Whisper, Radford et al., 2022)	Audio never leaves your infrastructure	Regulated data, or high volume
Judge with retrieval (after Lewis et al., 2020)	The judge also retrieves policy or product docs, and those quotes are verified too	When claims depend on rules outside the conversation
Human in the loop	Reviewers accept or reject each kept claim; rejections tune the rubric	When scores affect people's pay or reviews
Batch over many calls	Drop counts are aggregated into a live hallucination rate	Monitoring a model or prompt change over time

Pitfalls

Pitfall	What you'll notice	Remedy
Translating before judging	Quotes don't match the transcript, and politeness and code-switching are lost	Judge in the original language; translate only the glosses shown to the reader
Exact-match checking	Most true claims get dropped over punctuation or filler words	Fuzzy-match normalised text, with a similarity threshold
Ignoring time	A short, common quote ("yes") verifies against the wrong moment	Accept only the occurrence nearest the claimed time
Over-strict normalisation	Unrelated words match in scripts like Devanagari	Strip accents from Latin letters only; keep other combining marks
Scoring what never came up	Low scores for topics the call never touched	Let the judge abstain with a null score per category

Go further

• See it built: Voice Insight, an open-source demo of this pattern on call-centre quality review.
• Read: Judging LLM-as-a-Judge (Zheng et al., 2023) on where LLM judges go wrong.
• Next: CoALA, for the vocabulary used to map each stage.