Skip to content
AI360XpertBeta

Automatic Speech Recognition (ASR)

You talk, it types. Speech recognition slices your audio into tiny frames, scores each one for likely sounds, then merges the repeats into text.

ASR slices speech into short spectral frames, scores each frame for characters, and merges repeats into text
ASR slices speech into short spectral frames, scores each frame for characters, and merges repeats into text

Why Does This Exist?

Your voice is the fastest interface you own, and the messiest one for a computer. What reaches the microphone is just air pressure going up and down, sampled into numbers. Nothing in those numbers says where one word ends and the next begins. Say "I'm going to" fast and it lands as something like "I'm gonna": vowels smear, consonants vanish, and your accent, your speed, and the room all leave fingerprints on the wave.

That gap, between a continuous wave and discrete words, is the whole job. Automatic Speech Recognition (ASR) turns speech audio into text, and everything voice-controlled stands on it: assistants, captions, dictation, voice search.

A little grounding first, since the rest of the page assumes it. Sound becomes numbers through sampling: a microphone measures air pressure thousands of times a second, so even one second of speech is already thousands of numbers. The full treatment, sample rates and why they matter, lives in audio preprocessing.

This page covers the core job: wave in, text out. The architecture that reads whole clips before writing a word has its own page in encoder-decoder ASR. The search strategy that picks the final transcript lives in beam search. And the reverse trip, text back to voice, is text to speech.

Think of It Like This

Pausing a voice note to take dictation

A friend sends you a rambling two-minute voice note and you need it as text. Your plan: play it, pause every split second, jot down the likeliest syllable you just heard, then keep going. A long "oooh" stretches across five pauses, so you write it down five times, then clean up afterward: squash the repeats, cut the silences, and the sentence appears.

That's the job in miniature. The pauses are frames, the jotted syllables are per-frame scores, and the cleanup pass is the collapse step.

Where it stops: you cheat constantly. You know English, so you hear a mumble and guess the word from context. The collapse step knows nothing. It only deletes repeats and blanks. All the language smarts live in the scores and the decoder, never in the cleanup.

How It Actually Works

Three moves, in order: slice the wave into short frames, picture each frame as frequencies, then score every frame and collapse the repeats.

Why not feed the raw wave straight in?

Count it out. One second of audio sampled 16,000 times is a list of 16,000 numbers. Write the nnth number as x[n]x[n], where nn is just the position in the list. Ten seconds is 160,000 numbers in a row, and neighboring numbers barely differ while meaning lives hundreds or thousands of steps apart. That combination, very long and very fast-changing, is exactly what sequence models choke on. So nobody feeds the raw list in. Everybody repictures it first.

From wiggles to pictures

Cut the wave into overlapping slices. The standard slice is 25 milliseconds wide, and at 16,000 samples a second that is 0.025×16000=4000.025 \times 16000 = 400 numbers per slice. Start a new slice every 10 milliseconds, 160 numbers later, and one second gives 1+⌊(16000−400)/160⌋=981 + \lfloor (16000 - 400)/160 \rfloor = 98 slices, about a hundred frames a second. Call the frame index tt.

For each slice, ask how much energy sits at each pitch, low rumble versus high hiss. That measurement is a frequency transform with pitch spacing warped to match human hearing, the mel scale, and loudness compressed with a logarithm. The result per slice is one column of about 80 numbers, the mel bands, and the columns side by side form the mel-spectrogram: time along one axis, frequency along the other. One second of speech is now an 80×9880 \times 98 picture instead of a 16,000-long list. Networks that read pictures suddenly apply, which is the whole point of the detour. The tokenized alternative, learned audio codes instead of spectra, is neural audio codecs.

Scores per frame, then collapse

An acoustic model, usually a Transformer, reads that picture and scores every frame. For each frame tt it outputs a probability distribution P(c∣t)P(c \mid t): cc is one character from the vocabulary VV, tt is the frame, and the numbers across cc sum to 1. VV holds letters plus one extra symbol, the blank, written here as _\_. The blank means "no character here," and it earns its keep in the example below.

Why need a blank at all? People stretch sounds. Say "hi" slowly and the "h" fills two frames, silence fills one, "i" fills two. The frame-by-frame winners read [h,h,_,i,i,_][h, h, \_, i, i, \_], six labels for two letters. The collapse function BB fixes that with two dull rules: merge runs of the same label, then delete blanks. [h,h,_,i,i,_][h, h, \_, i, i, \_] becomes [h,_,i,_][h, \_, i, \_], then "hi". This is CTC, Connectionist Temporal Classification: independent per-frame scores plus a collapse that absorbs timing.

Two honest notes before the numbers. First, CTC assumes frames are independent given the audio, which is false and useful: it trains simply and streams naturally, but it can't use future context the way a full-clip reader can. When accuracy matters more than latency, readers that see the whole clip first win, and that design is encoder-decoder ASR. Second, the collapse assumes one voice per clip. Overlapping speakers break it, and no tuning fixes that. Split the speakers first.

Worked example: six frames spell "hi"

Six frames, three labels {h,i,_}\{h, i, \_\}, top probability each:

  • Frame 1: hh at 0.70
  • Frame 2: hh at 0.60
  • Frame 3: blank at 0.80
  • Frame 4: ii at 0.75
  • Frame 5: ii at 0.55
  • Frame 6: blank at 0.90

The greedy path takes the winner each frame: [h,h,_,i,i,_][h, h, \_, i, i, \_]. Its probability is the product:

0.70×0.60×0.80×0.75×0.55×0.90=0.12474≈0.1250.70 \times 0.60 \times 0.80 \times 0.75 \times 0.55 \times 0.90 = 0.12474 \approx 0.125

Collapse it: merge the hh run and the ii run to get [h,_,i,_][h, \_, i, \_], strip the blanks, read "hi". Those are the numbers drawn in the diagram above.

And notice what the blank earns. Without it, [h,h][h, h] would merge to "h" and a genuine double letter could never survive: part of "hello" would read [l,l][l, l] and collapse to one ll. Real systems keep blanks between same-letter frames, then let a wider search pick the transcript instead of trusting the greedy path. That search is beam search with a language model.

Code

The collapse is a few lines, and running it is the fastest way to feel the blank rule:

def ctc_collapse(path: list[str], blank: str = "_") -> str:    collapsed: list[str] = []    for label in path:        if not collapsed or label != collapsed[-1]:            collapsed.append(label)  # merge runs, blanks included    return "".join(c for c in collapsed if c != blank)

print(ctc_collapse(["h", "h", "_", "i", "i", "_"]))  # -> hiprint(ctc_collapse(["h", "_", "h"]))  # -> hh

The first print gives the worked example's "hi". The second shows why the blank exists: separated by a blank, two hh frames survive as "hh" instead of merging. Without that blank there is no way to write "hello".

Watch Out For

Your 'hello' comes out 'helo'

The symptom is specific: doubled letters vanish while everything else looks right. The cause is greedy decoding. The frame winners really do contain [l,l][l, l] somewhere, and the merge rule honestly deletes one, because without a blank between them two frames mean one letter.

Don't trust the argmax path. Decode with beam search plus a language model, which keeps blank-separated hypotheses alive and prefers real words. If doubles still vanish after that, your blank symbol is starved in training, not your decoder.

It worked on your laptop, then died in the kitchen

The symptom is a transcript that was clean on headset audio and turns to soup on room audio: substitutions climb, words fuse. The cause is mismatch. The model learned one acoustic world, close mic, quiet room, one speaker, and you tested it in another, reverb, clatter, crosstalk.

Match the worlds: train and test on multi-condition audio and add front-end cleanup, and respect the one assumption nothing fixes. One clip gets one transcript here, so overlapping voices must be split into separate tracks before this machinery ever sees them.

The Quick Version

  • Automatic speech recognition turns a continuous pressure wave into discrete text, one transcript per clip.
  • One second at 16 kHz is 16,000 numbers, so ASR repictures it: 25 millisecond slices become columns of about 80 mel-frequency energies.
  • An acoustic model scores every frame for every character plus a blank, and CTC merges repeats and cuts blanks to reach the text.
  • Six frames reading h, h, blank, i, i, blank collapse to "hi" with probability about 0.125.
  • Greedy winners eat double letters, so real systems decode with beam search and a language model instead.
  • One clip means one voice: overlapping speech and heavy noise break the assumption, so split speakers before you transcribe.