Voice Agent Architecture
A voice agent keeps one live audio link open, listens and talks at the same time, and lets you cut in. It turns your speech into actions and answers back in speech.
Why Does This Exist?
You say no, drop me at the north gate while the bot is still reading the old address, and it talks straight through you. You repeat yourself. It answers the first address anyway. You hang up. That is the failure this architecture exists to fix.
The obvious build chains three models in a row. Automatic speech recognition turns your voice into text, a text model picks a reply, and text-to-speech reads it aloud. Each stage waits for the last one to finish, so pauses stack. The middle text step also throws away how you sounded: rush, doubt, a joke, shouting behind you. And because the line is only half open, you can't cut in and it can't hear you pause. Feels robotic. Breaks on the first correction.
This page is about the live loop around the model: one open link, a way to hear when you stop, tool calls, and a reply that streams while it still listens. It does not re-teach the halves. For sound to text read automatic speech recognition, for text to sound read text to speech, for the direct audio engine that skips text read speech-to-speech models, and for how sound becomes tokens read neural audio codecs.
You need one idea to start. Speech can be chopped into short frames, and those frames can be scored as sounds or squeezed into tokens, much like words. Recognition maps those frames to words. Synthesis maps words back to waves. The pages above carry the full treatment. Here you only need the shape: audio in, meaning out, audio back.
Think of It Like This
A phone call, not voicemail ping pong
You and a friend swap voicemails. You record a full message, hang up, and wait. Your friend gets it written down, thinks, records a reply, and sends it back. Nobody can say wait, I meant Tuesday in the middle, because each message must finish before the next step starts. That is the cascade.
A voice agent should feel like a phone call instead. The line stays open both ways. You hear a short mm-hm while you talk. When you say hold on, not that gate, your friend stops mid sentence and listens. When they need to check something they say one sec and come back. The open line is the persistent link, hearing your breath pause is voice activity detection (VAD), deciding you are done is endpointing, saying hold on is barge-in, and the quick check is a tool call through tool use and function calling.
Where it stops: your friend hears meaning through a bad connection and remembers last week without trying. The agent hears only what its training covered, forgets unless you give it memory and a wrapper like agent harness design, and pays compute for every second the line stays open. Leave the line open in a noisy kitchen and it will try to answer the kettle.
How It Actually Works
A voice agent is four jobs on one open link: carry audio in, decide whose turn it is, think and act, and play audio back while still listening. The link is usually a persistent full duplex session, so either side can send at any moment. Full duplex means both directions stay live at once, unlike a request that goes one way and then waits.
The four parts of the loop
Carry audio in. Your mic frames go out as small chunks while you talk. The model starts reading before you finish, so most of your sentence is already understood when you stop. Short hops, no waiting for a full file.
Decide the turn. VAD marks each short frame as speech or not speech. Endpointing calls the turn over after a short hush. Set that hush too short and it cuts you off mid thought. Set it too long and every reply feels late. You'll tune this more than any other knob.
Think and act. Two builds fit here. A cascade runs recognition to text, then a text model, then synthesis. A direct build runs speech-to-speech prediction on audio tokens, so tone survives. Either way the model can call tools, check memory, and shape a reply. The trade is plain. The cascade keeps wording exact and logs clean, but drops tone and stacks waits. The direct path keeps tone and timing, but its long token strings fill context fast and exact wording is harder to pin down.
Play back while listening. Reply audio streams in chunks and plays as it lands. A small buffer holds audio that arrived early. If you cut in, the agent drops what is still buffered and listens. That drop is the whole feel of barge-in. No drop, no interruption.
When don't you want this? When you need an exact transcript, a searchable log, or word for word script following, or when calls are rare and cost matters more than feel, keep the text step or stay in text. Use the live loop when interruption, tone, and pace carry the task: address changes, tutoring, hands free help.
The loop rests on one assumption: its training covered your voices, accents, and noise, and your network and mic stay clean. When they don't, it mishears with confidence, copies the wrong style, or stutters on noise it never met.
How stopping and cutting in really happen
VAD is a small frame classifier. Speech, not speech, speech, hush. Endpointing watches that stream and fires when the hush runs long enough. Barge-in is the reverse path. While reply audio plays, inbound frames keep arriving. If loud sustained speech shows up, the agent stops playback, clears the buffer, and treats the new frames as the next turn.
Because speaker sound can leak back into the mic, you need echo cancellation plus a loudness gate. Without them the agent hears itself and stops itself. Test with the speaker on, not just with headphones. That's where most first demos fall over.
Worked example: 2200 ms against 620 ms
Take a teaching budget. These are assumed numbers so you can check the arithmetic, not measurements of any shipped product.
Let be the gap you feel, in milliseconds, from the moment you stop to the moment you hear the first reply sound. Let be transcription finish time, be time to the first text token, be time to the first playable audio chunk, be time to call the turn over, be time for the live model to return its first reply audio, and be one network hop for a chunk. Each is a stage wait in milliseconds. Each is a total gap in milliseconds.
The cascade waits in series:
Plug in the teaching values , , and :
The live loop overlaps listening with thinking, so it pays only the turn call plus the first audio plus one hop:
Plug in , , and :
The saving in this fixture is:
Barge-in saves more. Say the agent queued seconds of speech and you cut in after seconds. It drops what never played:
Same numbers sit in the diagram above: 900 plus 600 plus 700 into 2200 on top, 200 plus 400 plus 20 into 620 below.
Code
The point of this snippet is the check. Add the stage times for each build and you see where the wait lives. Change the inputs and the gap follows.
def cascade_gap(stt_ms: float, llm_ms: float, tts_ms: float) -> float: return stt_ms + llm_ms + tts_ms
def live_gap(vad_ms: float, first_audio_ms: float, hop_ms: float) -> float: return vad_ms + first_audio_ms + hop_ms
cascade = cascade_gap(900.0, 600.0, 700.0)live = live_gap(200.0, 400.0, 20.0)print(cascade) # -> 2200.0print(live) # -> 620.0print(cascade - live) # -> 1580.0The prints match the hand arithmetic above: 2200.0, 620.0, and 1580.0. Swap in your own measured stages and the same two functions tell you which stage to attack first.
Watch Out For
The agent keeps cutting itself off
It starts a sentence, stops, restarts, never finishes. Sounds like nerves. It is usually echo.
The cause is simple. Speaker audio leaks into the mic, VAD marks it as you speaking, and barge-in fires on its own voice. The symptom is unmistakable once you've seen it: the reply buffer clears every few words, only when the speaker is on, never on headphones.
Gate the mic before you tune the model. Turn on echo cancellation, require loud sustained inbound speech to trigger barge-in, and ignore the first fraction of playback while the echo path settles. Test with a laptop speaker at full volume in a hard room.
An open mic in a noisy room runs the bill and the nerves
Usage climbs all night, the agent answers the TV, and turns never end. The line stayed hot.
The cause is a missing gate on the client. Every cough, kettle, and background chat streams to the model, VAD stays lit, endpointing never fires, and you pay for audio nobody meant as a turn. The symptom in the logs is long sessions with almost no tool calls and lots of short confused replies.
Fix it at the edge. Run a small client side speech gate, auto mute on long silence, offer push to talk in noise, show a clear listening state, and cap turn length through your agent wrapper. If the room is loud, half duplex with a firm endpoint beats full duplex that never shuts up.
The Quick Version
- A voice agent is a live audio loop: stream speech in, track the turn, think and act, stream speech out while still listening.
- A cascade runs speech to text to speech in series and drops tone, while a direct model maps audio tokens to audio tokens and keeps it.
- Voice activity detection marks speech, endpointing calls the turn after a short hush, and barge-in clears the reply buffer so you can cut in.
- A teaching budget of 900 plus 600 plus 700 ms is 2200 ms for the cascade, against 200 plus 400 plus 20 ms or 620 ms live, saving 1580 ms.
- Keep the text step when you need exact wording, clean logs, or cheap rare turns, and use the live loop when tone and interruption carry the task.
- Gate the mic, cancel echo, and cap the turn, or noise and self echo will fire false cut ins and run the bill.