What Is Speaker Diarization: Who Said What in a Recording

Diarization in plain words

Speaker diarization answers the question "who spoke when". The input is a recording of a call or a meeting; the output is a list of segments: "Speaker 1: 00:00–00:12, Speaker 2: 00:12–00:31" and so on.

Note that diarization on its own does not know names. It groups similar voices and gives them generic labels. Turning "Speaker 1" into "agent Anna" takes an extra step — for example, knowing that the agent always speaks first, or comparing against a pre-recorded voice sample (that is speaker identification, a separate task).

How it works under the hood

A typical pipeline has several stages:

1. Voice activity detection (VAD). The system separates speech from silence, hold music and noise.
2. Segmentation. Speech is cut into short chunks of a fraction of a second to a few seconds.
3. Speaker embeddings. For each chunk a neural network builds a vector — a numeric "fingerprint" of timbre and speaking style.
4. Clustering. Similar vectors are grouped; each group is one speaker.
5. Boundary smoothing. Short false switches are removed and turn boundaries are refined.

Newer end-to-end models merge some of these steps into a single network and handle overlapping speech better, but the overall logic stays the same.

How diarization combines with transcription

Speech-to-text returns words with timestamps; diarization returns segments per speaker. They are aligned by time: each word gets the label of whoever was speaking at that moment.

Many cloud speech recognition services offer diarization as an option in the same request. There are also open-source tools you can host on your own servers when recordings cannot leave your infrastructure.

Where errors come from

Overlapping speech. When people interrupt each other, classic systems assign the chunk to one speaker or blur the boundaries. This is the most common problem in live meetings.

Similar voices. Two people of the same gender and age with a similar speaking style can merge into one speaker.

Mono audio. If a call is recorded in a single channel, the system can separate voices only by how they sound. If your telephony records in stereo (customer on one channel, agent on the other), you barely need diarization: roles are separated physically.

Unknown number of speakers. Without a hint about how many people take part, a model may split one person into two or merge two people into one.

Audio quality. Speakerphones, echo, street noise and heavy telephone compression all degrade the embeddings.

Short utterances. "Yes", "uh-huh", "got it" contain too little sound to identify a voice reliably.

How to improve accuracy in practice

• Record calls in stereo if your telephony supports it. It is the cheapest and most reliable improvement.
• Pass the number of speakers when you know it (an agent–customer call is almost always two people).
• Use separate microphones in meetings or record each video-call participant on their own track.
• Test on your own data. Take a few dozen real recordings, label them by hand and compare options using DER (diarization error rate — the share of time labeled incorrectly).
• Add role rules. For example, "the speaker who says the company greeting is the agent".

Where it is used

• Contact centers: script compliance, analyzing agent speech separately from customer speech, measuring talk-time ratio.
• Meeting minutes: automatic notes showing who took which action item.
• Interviews and podcasts: transcripts split by turn for editing.
• Legal and medical recordings: where it matters exactly who said a phrase (always with human review).