Speech-to-Speech Models
You speak and it answers in speech, with your sigh and its chuckle intact. No text step in the middle strips the tone away.
Why Does This Exist?
Say "sure, fine" through gritted teeth and you mean the opposite. A classic voice stack hears the words and misses the message. You'll feel talked over by a cheerful robot that never heard your sigh.
That stack is three models glued together. Speech recognition writes down what you said, a text model picks a reply, and speech synthesis reads it aloud. The middle text step keeps the words and drops everything else: pitch, pace, laughter, a shaky breath, background noise, two people talking at once. Delays add as well, because each stage waits for the last one to finish. You can't interrupt it and it can't read your pause.
A speech-to-speech model skips the middle text. It takes audio in and predicts audio out, so how you said it stays part of the input and how it answers stays part of the output.
This page covers that direct loop. For the halves it replaces, read automatic speech recognition for sound to text and text-to-speech for text to sound. For the compression trick that makes it possible, read neural audio codecs. You need one idea to start: sound can be quantized into discrete tokens, much like words, so a Transformer can predict the next sound the way it predicts the next word.
Think of It Like This
Passing notes versus talking face to face
Imagine you argue with a friend through two messengers carrying notepads. You speak. The first messenger writes "sure, fine" and runs it over. Your friend reads the flat words, writes "great!" and sends it back. The second messenger reads it aloud in a bright voice. Nobody heard your sigh. Nobody could cut in, because each note had to be finished and carried before the next step started.
Now sit across the table and talk. You hear the sigh before the sentence ends. You can lift a hand and interrupt. You answer softly because they sound upset. The messengers are the cascade and the table is the direct model: audio in, audio out, with timing and tone intact.
Where it stops: you understand sarcasm and noise you have never heard before. A direct model keeps only what its training taught it to keep. It still mistakes a cough for a laugh, and it copies your whisper when it should answer clearly.
How It Actually Works
A direct model treats speech as another language it can read and write. The pipeline has three parts: compress sound to tokens, predict reply tokens with a Transformer, expand reply tokens back to sound.
From waveform to tokens
Call the microphone signal , where counts samples in time. Call the sample rate in samples per second, and the clip length in seconds. Then the total sample count is . Raw is far too long to predict sample by sample, so an encoder compresses it first.
The encoder outputs frames at a rate , frames per second, so the frame count is . Each frame is quantized with codebooks, where a codebook is a fixed list of allowed sound pieces. Write the token at frame from codebook as . Call the codebook size, the number of choices per codebook. The full utterance becomes discrete tokens. That is the input the language model reads, and the details live in neural audio codecs.
One Transformer predicts sound with sound
The model holds one joint vocabulary: text pieces plus the audio choices per codebook. At each step it outputs a distribution , which reads as the odds for the next audio piece given all earlier pieces and the earlier codebooks in this frame. Sampling that distribution picks the next piece, the way decoding strategies pick the next word. Text can sit in the same sequence as helper tokens, but the path from your voice to its voice never has to collapse to words alone.
On the way out a vocoder turns reply tokens back into a waveform. Because the model predicts acoustic pieces directly, it can place a chuckle, a pause, or a stressed syllable where a text reply could only write "ha ha" or add an exclamation mark.
Why tone and interruption survive
Two properties fall out of that design.
First, prosody has somewhere to live. Pitch slides, speaking rate, and breathiness change the frame sequence, so they change the token sequence. A sarcastic "sure" and a sincere "sure" look different before any word is recognized. The model can learn that difference the same way it learns that two wordings differ.
Second, the model can listen while it talks. A cascade must finish transcribing before it starts thinking, then finish thinking before it starts speaking. A streaming direct model reads frames as they arrive and writes reply frames as soon as it has enough context, so either side can barge in. That is why direct systems fit voice agent architecture for live calls.
The trade is real. Audio token strings run much longer than their transcripts, so context fills fast and compute rises. Exact wording gets harder to pin down, and grading a voice answer takes listening, not string match. When you need an exact transcript, a searchable log, or word-for-word script following, keep the text step and use the cascade. Use direct speech when tone, timing, and interruption matter more than exact words.
The assumption underneath: your training pairs cover the voices, accents, and noise you will meet. When they don't, the model mishears or copies the wrong style with full confidence.
Worked example: 2 seconds become 400 tokens
Take a small teaching codec. Suppose seconds, samples per second, frames per second, codebooks, and choices per codebook. Nothing here describes a shipped product. It is a fixture you can check by hand.
Total samples:
Frames, each covering seconds, or 20 milliseconds:
Audio tokens:
Each choice among 1024 needs bits, since . So the coded size is:
Uncompressed speech at 16 bits per sample would be:
Compression ratio:
So the Transformer reads 400 tokens instead of 32000 samples, a 128 to 1 squeeze, and each token still carries enough acoustic detail for the model to hear a laugh as different from a word. Those are the same numbers drawn in the diagram below.
Code
This snippet recomputes the fixture above. Run it to check the diagram before you trust any codec setting of your own.
def codec_tokens(duration_s: float, frame_rate_hz: float, codebooks: int) -> int: frames = int(duration_s * frame_rate_hz) return frames * codebooks
def coded_bits(tokens: int, codebook_size: int) -> int: import math
bits_per_token = int(math.log2(codebook_size)) return tokens * bits_per_token
def compression_ratio(duration_s: float, sample_rate_hz: float, bits_per_sample: int, coded: int) -> float: pcm_bits = int(duration_s * sample_rate_hz) * bits_per_sample return pcm_bits / coded
tokens = codec_tokens(2.0, 50.0, 4)print(tokens) # -> 400
bits = coded_bits(tokens, 1024)print(bits) # -> 4000
print(compression_ratio(2.0, 16000.0, 16, bits)) # -> 128.0The prints say 400 tokens, 4000 coded bits, and a ratio of 128.0. If your codec uses a different frame rate or codebook count, change those three inputs and compare before you size your context window.
Watch Out For
It mirrors your voice when it should not
Whisper a question in a noisy room and the answer comes back as a whisper you can barely hear. Sound angry and it answers angrily, which reads as escalation. The symptom is style copying that feels empathetic in a demo and wrong in production.
The cause is direct: acoustic style is in the token stream, so the model treats your delivery as the requested delivery. Fix it with an explicit output style in the system instruction, such as a fixed voice and pace, and test flat versus emotional inputs on purpose. For sensitive flows, lock the reply voice to one preset instead of letting it continue your tone.
It answers the first half of your sentence
You say "book a flight to... um... Paris" and it starts booking before you finish, or a short pause triggers a full reply. The symptom is confident answers to incomplete requests, plus constant interruptions in noisy rooms.
The cause is turn taking, not intelligence. A streaming model decides on its own when you are done, and a short silence or filler word looks like the end. Fix it by tuning endpointing: require a longer silence before replying, show a visible listening state, and keep a push to talk or hold to talk option where mistakes cost money. Log barge in rate alongside answer quality so you can see the trade.
The Quick Version
- A speech-to-speech model maps input audio tokens straight to output audio tokens, with no text bottleneck in between.
- The middle text step of the old cascade keeps words and drops pitch, pace, laughter, and overlap, while delays stack because stages run in series.
- A neural audio codec makes the direct path possible by compressing each 20 millisecond frame into a few discrete codebook tokens.
- In the teaching fixture, 2.0 seconds at 50 frames per second with 4 codebooks give 100 frames and 400 tokens, a 128 to 1 squeeze over 16 bit PCM.
- Pick the cascade when you need exact transcripts or word-for-word control, and pick direct speech when tone, timing, and interruption decide whether the call works.