Speaker Diarization
One recording, several voices. Diarization writes the speaker labels in the margin: who spoke, and exactly when.
Why Does This Exist?
You run recognition on a one-hour panel with four guests. The transcript comes back complete and useless: every word is there, and none of them has a name on it. Who made the claim at minute 23? Was that interruption a joke or an objection? A wall of text cannot tell you, and labeling an hour by hand is slow, boring, and full of timestamp drift.
Speaker diarization is the step that writes names in the margin. It cuts the recording into segments, groups the ones that sound like the same person, and stamps each group with a speaker label plus start and end times. Without enrollment recordings it says Speaker A and Speaker B rather than real names. That is still enough to print a readable transcript, to measure talk time, or to hand the next system clean turns to work with.
This page covers only the who-spoke-when step. How the words themselves get recognized lives in automatic speech recognition, the transcription engine most teams pair it with is Whisper architecture, and one system that consumes clean turns is voice agent architecture. The one prerequisite is short: recognition maps sound to words, diarization maps sound to speaker labels, and production joins the two on timestamps.
Think of It Like This
Sorting unsigned notes by handwriting
Someone slides a stack of unsigned meeting notes across your desk. No names, just handwriting. You start sorting: the loopy slant goes left, the cramped print goes right, the left-hander's smudge makes a third pile. Three piles, three writers. You still do not know their names. Then the organizer points at the piles and says the left one is Priya, and every note in that pile is solved at once.
That is the whole mechanism. Each note is a short audio window, the handwriting is the voiceprint, each pile is a speaker, and the organizer naming the piles is enrollment. Without enrollment you get Speaker A and Speaker B, which still answers who spoke when.
Where it stops: handwriting sits still on paper, but voices wobble. A cold, a laugh, a cheap mic, or a walk across the room all change the sound without changing the speaker. And if two people write on the same note at once, your one-note-one-writer rule breaks. Overlapping speech breaks diarization the same way, which is why overlap gets its own warning below.
How It Actually Works
You hand the system one mixed recording. It hands back labeled segments: who spoke, from which second to which second. Four stages do it: cut silence, fingerprint voices, group fingerprints, fuse with words.
Cut speech from silence
Voice activity detection (VAD) marks which frames hold speech. Call the raw recording and the kept speech windows . In plain words: is the whole file going in, each is one short slice of speech, and is just how many slices survived. Slices run around one to two seconds, stepped so neighbors overlap. Silence, music beds, and mic bumps get dropped here, which saves the expensive steps for frames that matter. Set the detector too strict and it eats soft consonants at turn edges. Set it too loose and you pay to fingerprint noise.
Turn each window into a voiceprint
A neural embedding model maps each window to a speaker embedding . In plain words: is the fingerprint reader, is the slice going in, and is the list of numbers coming out that describes the voice and ignores the words. Training pushes windows from the same voice close together and windows from different voices far apart, even when the text matches. Your laugh and your whisper land near each other. The same sentence from two hosts lands far apart. That word-independence is the property everything after depends on. The full treatment of the object itself is in embeddings, and the short version is embedding.
Length-normalize each voiceprint so . In plain words: is the vector's length, and forcing it to 1 means only direction carries identity, never loudness. Then closeness is the cosine:
Here is the dot product, the sum of pairwise products, and is the similarity score between windows and . Read it as an angle: means identical direction, means at right angles, means opposite. How angles become keep-or-split decisions is in similarity metrics, with the formula itself in cosine similarity. Grouping nearby vectors without labels is clustering.
Score pairs and form speakers
Clustering joins windows whose score clears a threshold (tau, the cutoff, for example ) and splits the rest. Write the answer as labels , where each is the speaker tag for window , taking values like , , or . Nobody hands the system the speaker count. The number of groups is discovered from the data. A re-segmentation pass then sharpens the rough boundaries, and the labels fuse with recognized words on timestamps to produce lines like "Speaker B: how are you?".
The trade is window length. Short windows catch fast interruptions, but each voiceprint is thin and noisy. Long windows give fat, stable voiceprints, but they smear over quick back-and-forth. When the recording holds one person reading aloud, skip this step entirely: there is nothing to separate. When you already hold voiceprints for known people, check new audio against those prints instead of blind clustering, because blind groups can only return letters, never names.
This design assumes one speaker per window. Overlap violates it, and the warnings show what that costs.
Worked example: three windows, two voices
Say three kept windows produce voiceprints , , and . Think of and as two turns from one host and as the co-host answering.
First check the lengths so the cosine means what you think. and by inspection. For :
Close enough to 1 that direction does the talking. Now the dots: , , and . Divide by the lengths:
Set . Only the pair clears it, so the labels come out . The diagram draws exactly this: the two dots sit inside one boundary with marked keep, while the pair splits. The snippet below prints these same numbers.
Score a whole meeting with the diarization error rate (DER):
is false-alarm time, silence labeled as speech. is missed-speech time, speech labeled as silence. is confusion time, the right speech with the wrong speaker. is all reference speech time. Read the parts separately: high blames your detector, high blames your voiceprints or your cutoff.
Code
The point here is the decision rule, not the formula retyped: normalize, score with cosine, group above the cutoff. Three windows, same numbers as the worked example.
import numpy as np
def cosine(a: np.ndarray, b: np.ndarray) -> float: return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
a = np.array([1.0, 0.0])b = np.array([0.92, 0.39])c = np.array([0.0, 1.0])tau = 0.70
print(round(cosine(a, b), 3)) # -> 0.921print(round(cosine(a, c), 3)) # -> 0.0print(round(cosine(b, c), 3)) # -> 0.39print(cosine(a, b) > tau) # -> TrueSame verdict as the hand computation: and group, splits. Pick on held-out audio from your own setup, never by guessing. A meeting room and a phone call want different cutoffs.
Watch Out For
Overlap invents a third speaker
Two people talk over each other for a second. That window holds both voices, so its voiceprint lands halfway between them. Clustering sees a stranger and opens a phantom Speaker C that only exists during interruptions.
The symptom is unmistakable: brand-new speaker labels that last a second or two and sit exactly on crosstalk. The fix is to catch overlap before clustering. Run an overlap detector, keep overlapped frames out of the grouping step, then assign each one to the nearest clean speaker afterward. For heavy crosstalk, separate the sources first. Never trust a new speaker built only from interruption frames.
One voice splits in two when the channel shifts
The same host is Speaker A before a mic move and Speaker C after it. Or two similar voices melt into one label for the whole call. The cause is the cutoff, not the concept: too high splits, too low merges, short windows add noise, and far-field audio never matches close-mic tuning.
The symptom is identity drift without new people entering. The fix is to tune on held-out audio recorded the same way, prefer slightly longer windows plus a re-segmentation pass, and read the DER parts instead of one number. When names matter, enroll the speakers up front rather than hoping blind letters stay stable.
The Quick Version
- Speaker diarization answers who spoke when. Recognition answers what was said. A readable multi-speaker transcript needs both, joined on timestamps.
- Each short window becomes a length-normalized voiceprint that describes the voice and ignores the words.
- Cosine similarity scores pairs of voiceprints. Pairs above the cutoff share a label, and pairs below it split.
- The speaker count is discovered by clustering, never given. Without enrollment the answers are letters like A and B, not names.
- Overlap breaks the one-window-one-speaker assumption and invents phantom speakers unless overlapped frames get special handling.
- Tune the cutoff and the window length on audio from your own setup. Short windows catch interruptions, while long windows give stabler voiceprints.