Gen AI

Speech-to-Speech Models (S2S)

Instead of converting your voice to text, processing the text, and generating a new voice, true Speech-to-Speech models process audio directly. This allows them to hear your sarcastic tone or nervous laughter and respond with the appropriate emotion.

Speech-to-Speech models process audio directly into audio without converting to text first, preserving emotion, tone, and timing.

Why Does This Exist?

If you used a voice assistant like Alexa, Google Assistant, or early ChatGPT Voice before 2024, you were actually using a Cascaded Pipeline, not a true audio AI.

If you sighed heavily and said, "Sure, I guess that's fine," in a very sarcastic tone, the system would completely misunderstand you. Here is why:

ASR Step: The Speech-to-Text model converts your audio into text: "Sure, I guess that's fine."
The Text Bottleneck: All information about your heavy sigh and sarcastic tone is permanently lost.
LLM Step: The text LLM reads the text, assumes you are genuinely agreeing, and outputs: "Great! I'm glad you're happy."
TTS Step: The Text-to-Speech model reads that response in a cheerful, upbeat robot voice.

Because text is a massive "bottleneck," the AI was effectively deaf to emotion, tone, breathing, and background noise. Direct Speech-to-Speech (S2S) Models (like GPT-4o) were invented to eliminate the text bottleneck entirely.

Think of It Like This

The Email vs. The Phone Call

Cascaded Pipeline (Email): You dictate a message to your secretary, who types it out and emails it to a friend. Your friend reads the plain text email and dictates a response to their secretary. You miss all the nuance, laughter, and tone of voice.
Speech-to-Speech (Phone Call): You call your friend directly. They hear you laugh, they hear when you take a breath, and they can interrupt you immediately if they have a thought. The communication is rich, instantaneous, and emotional.

How It Actually Works

To build a true S2S model, researchers had to train LLMs on raw audio tokens from the very beginning.

1. Joint Training on Audio and Text

Using Neural Audio Codecs, we compress audio into discrete tokens (just like text). We take a massive LLM and train it on a mixture of text tokens and audio tokens simultaneously. The LLM learns that the audio token for a "sigh" often precedes sentences expressing frustration. It learns that if someone is speaking quickly (represented by dense, rapid audio tokens), they might be panicked.

2. Audio-In, Audio-Out Inference

When you speak to GPT-4o, the raw audio wave is converted directly into audio tokens and fed into the neural network. There is no intermediate text. The LLM processes your tone, your hesitations, and the background noise. It then outputs audio tokens directly. Because it is generating audio natively, it can choose to generate the audio tokens for a chuckle, a whisper, or an excited shout.

3. Extremely Low Latency

A cascaded pipeline is incredibly slow. You have to wait for the ASR to finish transcribing, then wait for the LLM to finish generating text, then wait for the TTS to generate the audio. This often resulted in a 3 to 5-second delay, making conversational back-and-forth impossible. Direct S2S models are typically Streaming. As you speak, the model is constantly updating its internal state. The moment you stop speaking, it instantly begins streaming audio tokens back to you. This drops the latency to around 300 milliseconds, identical to human reaction time in a normal conversation.

Show Me the Code

This pseudocode highlights how a unified Multimodal LLM skips the text steps entirely.

1import torch2
3def direct_speech_to_speech(multimodal_llm, user_audio, codec):4    """5    True S2S skips ASR and TTS entirely.6    """7    # 1. Convert user's continuous audio wave into discrete audio tokens8    # No text transcription occurs!9    input_audio_tokens = codec.encode(user_audio)10    11    # 2. Feed the audio tokens directly into the LLM12    # The LLM understands the acoustic features natively13    generated_audio_tokens = []14    15    for _ in range(MAX_LENGTH):16        # The LLM outputs the next AUDIO token, not a text token17        logits = multimodal_llm(input_audio_tokens, generated_audio_tokens)18        next_audio_token = sample(logits)19        20        generated_audio_tokens.append(next_audio_token)21        22        if is_end_of_turn(next_audio_token):23            break24            25    # 3. Decode the LLM's raw audio tokens back into sound waves26    output_audio = codec.decode(generated_audio_tokens)27    28    return output_audio

Watch Out For

The Hallucination of Emotion

Because S2S models are trained to mimic human conversational patterns, they will sometimes generate "filler" sounds like "um," "uh," or nervous laughter even when it makes no logical sense for an AI to do so. Furthermore, if you whisper to the model, it will often whisper back. While this feels empathetic, it can occasionally lead to unsettling or inappropriate interactions if the model misinterprets your acoustic tone and responds with unwarranted intimacy or aggression.

The Quick Version

Before 2024, voice assistants used a Cascaded Pipeline (ASR $\rightarrow$ Text LLM $\rightarrow$ TTS), which destroyed all emotional and acoustic information by forcing everything through a "text bottleneck."
Direct Speech-to-Speech (S2S) models (like GPT-4o) process audio natively. They ingest audio tokens and output audio tokens.
Because they process audio directly, they can hear tone, emotion, and background noise, and they can generate responses with sighs, laughter, and specific vocal intonations.
Skipping the text steps massively reduces latency, enabling real-time, instantaneous conversational AI with reaction speeds similar to human beings.

Related concepts

Vision-Language Models (VLMs)