Voice AI Is Becoming a Real-Time Systems Problem

Unlock Exclusive Savings for Your Company’s Travel & More

Create a new Corporate Account between 9/1/25 - 3/31/27 and receive up to 10% off eligible Enterprise Profile rides taken in the U.S. (max $10 off per trip; 5 rides per user) for Authorized Users for 90 days or until 3/31/2027, whichever occurs first. Valid only for new Uber for Business clients in the U.S. that sign up here.

Promotion valid until 3/31/2027. Subject to change or cancellation at anytime. Other terms and conditions apply, see full terms.

Get Started Today

For years, voice assistants have followed a familiar sequence: the user speaks, the system detects the end of speech, the speech is converted to text, a language model generates an answer, and another model converts the answer back to audio.

Each stage can be impressive individually, but the combined experience often feels mechanical. The assistant waits too long, interrupts at the wrong moment, ignores vocal cues — or makes awkward pauses before responding.

The latest generation of sound systems seeks to get rid of these basic limitations. Instead of treating a conversation as a series of audio requests, it treats speech as a continuous two-way stream. Models can receive audio while producing audio, determine whether to listen or respond, and delegate complex work to other models without interrupting the conversation.

This is a more profound change than faster speech generation. Voice AI is evolving from chatbots with microphones to real-time distributed systems.

The real problem was never only speech quality

Human conversation takes place over a very demanding period of time. Studies of everyday interactions show the distance between speakers is about 200 milliseconds. People often start planning their responses before the other person has finished speaking, (in practice) using vocabulary, syntax, intonation, pacing. And context to anticipate when their turn will end.

Traditional voice channels work against this natural rhythm.

The speech-to-text component must wait for enough audio to produce a reliable transcription. The turn detector must determine whether a pause means the user is finished or just thinking. The language model must process the completed prompt. Text-to-speech should then create and start playing the response.

Even though each component is fast, serial execution will pile up latency. Transcription can also remove information conveyed through tone, hesitation, emphasis, laughter, volume — or tempo.

The live speech-to-speech model improves this by understanding and producing audio natively. OpenAI’s Realtime documentation says that this avoids the intermediate steps of speech-to-text and text-to-speech, reducing latency while preserving information about tone and inflection.

But direct audio processing alone does not solve all interaction problems. Speech-to-speech models that are still waiting for turn detectors remain stuck in the same basic structure: listen, stop, decide, respond.

The more important transition is from turn-based inference to continuous inference.

Full duplex removes the artificial boundary between listening and speaking

Full-duplex sound systems can receive and output audio At the same time,. Instead of viewing conversations as alternating messages, it observes the ever-changing flow and repeatedly decides what conversational actions are appropriate.

The action may be to continue listening, start answering, pause, greet the speaker, interrupt, yield, call a tool, or wait for a delegated result. GPT-Live is designed based on this full duplex model, eliminating a separate loop detector from the main audio path.

This is important because silence is ambiguous.

User can pause because it has finished. They may also search for a word, emphasize a point, check information, or wait for a reaction. Small detectors that only examine silence duration or acoustic features have limited context to differentiate them.

Full voice models can theoretically make decisions using a wider range of conversations. This can consider what was said, how it was said, whether the sentence sounded complete, whether the user seemed unsure — and whether acknowledgment would be helpful without taking over the conversation.

The interaction becomes less like sending voice commands and more like maintaining a shared conversational state.

The architecture separates talking from thinking

The main design decision was to divide the system into two paths.

The direct path handles work that cannot be postponed: receiving audio frames, performing continuous speech inference — and returning audio to the user.

Separate asynchronous paths handle tracing, tool calls, business logic, persistence, analytics, policy evaluation, and deeper reasoning. A slow external service may delay the results, but it doesn’t stop audio frames from moving through the direct path.

This resembles a pattern already used in high-performance transaction systems. The synchronous path contains only what must happen before the user can continue. Everything else is delegated to background processing.

The difference is that the deadline is now perceptual. Pending database writes may not be visible. Late audio frames can introduce pauses, (at least that’s the general idea) clicks, overlaps, or broken sentences that users hear immediately.

This separation creates an important product advantage: application behavior can change without destabilizing the media layer. Developers can add tools, modify policies, change databases, or introduce approval workflows while leaving the latency-sensitive audio transport relatively small and predictable.

OpenAI’s public Realtime architecture reflects the same principle through a sideband connection. The user’s client can communicate with the realtime session through WebRTC, while the application server connects separately to monitor the session, update instructions, handle tools, and keep sensitive business logic away from the client.