Voice & Real-Time Agent Design (2024–2026)

Parent: Global AI Hub Research Corpus · researched 2026-05-31· 1 source · 0 concepts

> Scope note: This report covers the application / design layer of voice agents — architecture choices, real-time APIs, turn-taking, latency engineering, orchestration frameworks, components, UX, and

Voice & Real-Time Agent Design (2024–2026): Research Report

> Scope note: This report covers the application / design layer of voice agents - architecture choices, real-time APIs, turn-taking, latency engineering, orchestration frameworks, components, UX, and evaluation. Model-internal speech architecture (Whisper encoders, the Thinker–Talker decomposition, audio tokenization) is deliberately out of scope and is covered by the existing multimodal-llm-architecture reference. Cross-reference that reference for "how the model produces audio"; this report is "how you build and operate a voice product around such models." [source]

Overview

A voice agent is a real-time conversational system that listens, reasons, and speaks back inside a turn-taking loop tight enough to feel human. The dominant 2024–2026 design tension is cascaded pipelines (STT → LLM → TTS as discrete, swappable stages) versus speech-native / speech-to-speech (S2S) models (one model that ingests and emits audio directly, e.g., OpenAI's gpt-realtime and Google's Gemini Live API). Cascaded wins on control, debuggability, compliance, and provider choice; S2S wins on latency and emotional prosody but is hard to audit and gate (Coval, Speko, Hamming). [source]
As of mid-2026, most production voice agents still use cascaded architectures, with S2S adoption projected below 15% in H1 2026 and rising to roughly 25–30% by end of H2 2026 as evaluation tooling and compliance frameworks mature (Coval). The engineering north star is a voice-to-voice latency budget under 800 ms, and the field has converged on streaming everything, semantic turn detection, and first-class barge-in as the table-stakes pattern (Hamming, Smallest.ai). [source]

1. Architecture: Cascaded vs. Speech-Native (S2S)

Cascaded = an orchestration layer coordinating specialized services: VAD → streaming STT → LLM → streaming TTS (Softcery). S2S = a single model that "processes the entire exchange in a single latent space with no intermediate representations" (Speko). [source]
Trade-off summary (confidence: High - multiple independent sources agree): [source]
When to use which: S2S for premium/empathetic experiences (mental-health, coaching, luxury, multilingual); cascaded for high-volume tier-1 support, regulated industries, and complex tool-driven workflows (Coval). A streaming cascade (Deepgram + GPT-4o-mini + Cartesia) can already hit sub-1-second latency, narrowing S2S's main advantage (Speko). [source]

2. Real-Time APIs

OpenAI Realtime API graduated to GA in 2025 with the gpt-realtime S2S model (GA snapshot gpt-realtime-2025-08-28, 32,768-token context, 4,096 max output) (OpenAI Realtime blog, OpenAI docs). Key design facts: [source]
Transport: WebRTC is recommended for client media (browser/mobile) for "more consistent performance"; WebSockets for server media pipelines (phone calls, broadcast ingest) (OpenAI docs, webrtcHacks). WebRTC sessions use the new https://api.openai.com/v1/realtime/calls endpoint. [source]
Auth: Ephemeral client secrets via POST /v1/realtime/client_secrets keep permanent keys off the client; the GA interface drops the OpenAI-Beta: realtime=v1 header (OpenAI Realtime blog). [source]
Sessions: Configured via session.update (instructions, turn-detection mode, truncation) (OpenAI Realtime blog). [source]
Tool use / function calling: Keep tools and business logic server-side via a sideband control channel - two connections to one session (one client, one server); the server monitors, updates instructions, and answers tool calls. GA adds asynchronous function calling with automatic placeholder responses ("I'm still waiting on that") to suppress hallucination during pending calls (OpenAI Realtime blog, OpenAI server-controls docs). [source]
Gemini Live API (GA on Vertex AI) is built on Gemini 2.5 Flash Native Audio - raw audio through a single low-latency model, with 30 HD voices in 24 languages, emotion-aware responses, live speech-to-speech translation, and tool use (function calling + Google Search) (Google Cloud blog, Gemini Live API docs, blog.google). It is multimodal - agents can converse about live visual streams (charts, video) alongside spoken input (Google Cloud blog). [source]

3. Turn Detection, Endpointing, and Barge-In

The hardest UX problem in voice agents is knowing when the user is done speaking and letting them interrupt. Three layers, increasing in sophistication (LiveKit, LiveKit docs): [source]
VAD (server VAD): Classifies each audio frame as speech/non-speech in real time (Silero VAD is the widely-used reference model). Cheap but dumb - it only sees energy, not meaning (LiveKit). [source]
Endpointing: Uses a silence-timeout on top of VAD to declare end-of-turn. The core latency trap: "A silence timeout set to 800 ms adds nearly a full second to every single response before the pipeline even starts" (LiveKit). [source]
Semantic / model-based turn detection: Reads the partial transcript and infers whether sentence structure is complete. "Can trigger before trailing silence occurs, which is the main latency advantage," letting silence thresholds drop to 200–300 ms without raising false interruptions (LiveKit, appinventiv). OpenAI exposes both Server VAD and Semantic VAD modes (OpenAI VAD docs); Azure offers a noise-resilient Semantic VAD (Microsoft). [source]
Two persistent VAD-only failure modes: false positives (pauses/hesitations read as turn ends) and delayed responses (strict silence thresholds stall replies after the user is semantically done) (LiveKit, SkyScribe). [source]
Model-integrated turn detection is the 2026 frontier: Deepgram Flux is a "conversational speech recognition" model with first-of-its-kind model-integrated end-of-turn detection, detecting end-of-turns in ~260 ms (median <300 ms, p95 ~1.5 s) at Nova-3 accuracy - collapsing ASR + VAD + endpointing into one model (Deepgram Flux intro, Deepgram Flux docs). [source]
Barge-in (interruption): Keep the turn-detection layer active even while the agent is speaking; on detected user speech, cancel the current TTS stream and hand control back to STT immediately. Echo cancellation runs client-side; push-to-talk is the fallback for devices lacking echo cancellation (LiveKit, futureagi). Backchanneling ("mm-hm", "yeah") is the subtle case: pure VAD misreads backchannels as either silence or a full barge-in, so the 2026 stack moves to dedicated turn-taking models that classify backchannel vs. barge-in as a learned signal (appinventiv, DEV/Mishra). [source]

4. Latency Engineering

Target: under 800 ms end-to-end for production voice agents (Hamming, Ultravox). Component latencies are cumulative and sequential - a naïve "each stage waits for the previous" pipeline easily exceeds 2–3 s (Smallest.ai). Representative budget (confidence: High - consistent across Hamming, Smallest.ai, Twilio): [source]
Sources: Hamming, Smallest.ai. [source]
Where it accrues: LLM inference is the dominant cost - Hamming attributes ~70% of total latency to it, "making model selection critical" (Hamming). For voice, TTFT matters more than total generation time, because TTS can begin streaming as soon as the first sentence is complete (Smallest.ai, Cresta). [source]
Mitigations: streaming STT (save 100–200 ms), streaming TTS (save 200–400 ms on TTFB), fast LLMs (GPT-4o-mini ~400 ms TTFT, Claude 3.5 Haiku ~360 ms), semantic endpointing (save 200–400 ms), and geographic colocation (US-coast-to-coast +60–80 ms, US–Europe +80–150 ms, US–Asia +150–250 ms) (Hamming, relinns). [source]

5. Components: Streaming ASR/STT and TTS

STT (May 2026 landscape): Deepgram leads on voice-agent latency and end-of-speech detection (Nova-3 hosted WER ~5.26% on its own real-world test set); Nova-3 + Flux is positioned as the May-2026 voice-agent default. ElevenLabs Scribe v2 Realtime leads multilingual real-time; AssemblyAI leads transcript intelligence; Whisper covers 57+ languages but is beaten by Scribe v2 on its 30 (futureagi STT, Softcery STT/TTS, AssemblyAI). [source]
TTS time-to-first-audio (TTFA) benchmark, May 2026 (confidence: Medium - single benchmark source, vendor-published): Gradium TTS 155 ms P50 (2 ms IQR); Cartesia Sonic-3 188 ms P50 but 100 ms IQR (50× wider); ElevenLabs Turbo v2.5 264 ms P50 (28 ms IQR), Flash v2.5 288 ms; Deepgram Aura-2 313 ms (Gradium benchmark). Positioning consensus: ElevenLabs = voice realism / cloning benchmark; Cartesia = streaming-latency benchmark (futureagi TTS, Cartesia, Cekura TTS). [source]

6. Voice UX / Design

Voice UI's defining constraint: nothing is visible and the spoken sentence is gone the moment it lands, so error recovery is a primary discipline, not an edge case (InfoWorld, fuselabcreative). Key patterns: [source]
Confidence-tiered confirmation: High confidence → act + implicit confirmation ("I've sent the invoice to your inbox"); medium confidence → clarify ("I found 3 contacts named John - which one?"); low confidence → graceful fallback (InfoWorld). Implicit confirmation beats explicit yes/no questions in enterprise flows. [source]
Error recovery as trust: users forgive the first error; doubt by the second; "doesn't work" by the third - so agents must state when confused and offer concrete next steps, not cryptic failures (InfoWorld, Clearly Design). [source]
Consent before consequential action: the foundational moment before an agent takes a significant action (Smashing). [source]

7. Evaluation

A 4-layer voice-agent quality framework: (1) Infrastructure - audio quality, latency, ASR/TTS perf; (2) Execution - intent classification, response accuracy, tool-calling logic; (3) User-behavior - interruption handling, conversation flow, sentiment; (4) Business-outcome - containment rate, first-call resolution (FCR), escalation (dev.to/Paul, Cekura metrics, Hamming metrics). [source]
Headline metrics and targets: WER = (S+D+I)/total words, target <5% for enterprise - but WER "ignores interaction dynamics" (barge-in, endpointing, turn-taking) and underestimates real-world UX under noise/accents/far-field (dev.to/Paul). Interruption handling: agent should stop within 200 ms of user speech and acknowledge/address the interruption >90% of the time (dev.to/Paul). MOS (naturalness) ~4.5/5 for near-human; Task Success Rate / FCR target ~85%+ (dev.to/Paul, Hamming testing, Braintrust). Testing tooling (Hamming, Coval, Cekura, Braintrust) emphasizes simulated callers / regression / load testing before live traffic, and per-component instrumentation tagged by call_id/turn_id (Hamming testing, Softcery QA). [source]

Tools / Frameworks

Orchestration frameworks (open source): [source]
Pipecat - Python, pipeline/frame model stringing VAD → STT → LLM → TTS; "most elegant" for 1:1 voice assistants; transport-agnostic (WebSocket, WebRTC via Daily, Twilio media streams) but you assemble production deployment yourself (f22labs, Cekura framework, AssemblyAI frameworks). [source]
LiveKit Agents - WebRTC-first; ships production transport (rooms, participant mgmt, track routing, egress/ingress, recording, SFU) in Go/Python/Node; best for multi-participant and latency-sensitive deployments that also need SIP (Mansoori, LiveKit GitHub, WebRTC.ventures). [source]
TEN - third open-source contender alongside LiveKit/Pipecat (Medium/Garcia). [source]
Managed platforms (build-faster, telephony-included): [source]
Vapi - middleware/BYO-everything (your LLM, TTS, STT, telephony Twilio/Vonage/Telnyx); ~500–800 ms latency; weaker native telephony (no warm transfer / branded calls / native SIP) and gated compliance (Retell vs Vapi, SuperDupr). [source]
Retell AI - managed, voice-quality + sub-500 ms latency leader; ships warm transfer, branded calls, native SIP trunking, KB retrieval, HIPAA/SOC2/GDPR on every plan (Retell vs Vapi, ainora). [source]
Bland AI - API-first, high-volume outbound; ~600–900 ms latency, all-inclusive pricing, strong data governance (SuperDupr, ainora). [source]
Telephony integration: PSTN reaches a WebRTC room via SIP trunking - buy a number (Twilio/Vonage/Telnyx), point its Voice URL at the framework's SIP URI. LiveKit telephony supports DTMF, call transfers, secure trunking, HD voice, region pinning, noise cancellation, plus connectors that bridge Twilio/WhatsApp without manual SIP config (LiveKit telephony docs, LiveKit agents-integration). [source]

Practical Patterns

Stream every stage; never block. Begin TTS on the first complete sentence of the LLM stream; begin LLM on the first finalized STT segment. Naïve sequential pipelines hit 2–3 s (Smallest.ai, Hamming). [source]
Optimize for LLM TTFT, not total tokens. It's ~70% of the budget; pick a fast tier model and keep the prompt small (Hamming). [source]
Use semantic (or model-integrated) turn detection to push silence thresholds to 200–300 ms without false cut-offs; Deepgram Flux folds end-of-turn into ASR (~260 ms) (LiveKit, Deepgram Flux). [source]
Make barge-in first-class. Keep turn detection live during playback; cancel TTS within 200 ms; distinguish backchannel from interruption with a turn-taking model (LiveKit, appinventiv). [source]
Keep tools/business logic server-side via a sideband channel (OpenAI Realtime) so secrets and policy never touch the client (OpenAI Realtime blog). [source]
Match transport to context: WebRTC for browser/mobile clients; WebSocket for server-side / telephony media (OpenAI docs). [source]
Tier confirmations by confidence and treat error recovery as a primary design surface (InfoWorld). [source]
Instrument per component (call_id/turn_id), watch p95 tail latency, and gate releases with simulated-caller regression tests (Hamming testing, appinventiv). [source]
Default cascaded for regulated/tool-heavy; reserve S2S for empathy-first experiences, or use a hybrid (S2S for conversation, cascade for compliance-gated branches) (Coval). [source]

Anti-Patterns

Blocking pipeline - waiting for the full LLM response, then starting TTS, then playing audio; adds hundreds of ms per turn for no reason (appinventiv). [source]
Over-long silence timeout - an 800 ms endpoint timeout adds ~1 s to every turn (LiveKit). [source]
Endpointing that holds too long - semantic detection waiting past the fallback silence timeout shows up as a discrete tail-latency jump, not a gradual rise (appinventiv). [source]
No / poor barge-in - speaking over the user or ignoring interruptions; treating backchannels as full barge-ins (futureagi, appinventiv). [source]
Retry cascades without hard timeouts - a 30 s API outlier strands the caller even if it averages away in dashboards (appinventiv). [source]
Weak multi-turn context - failing to resolve "cancel that one" three turns deep; demos pass, production fails (appinventiv). [source]
Deploying S2S into regulated flows - no text intermediary means non-compliant audio can reach the customer before any filter (Coval). [source]
WER-only evaluation - ignores barge-in, endpointing, turn-taking, and real-world noise/accents (dev.to/Paul). [source]

Suggested Sub-Concepts (future child concepts)

Cascaded vs. speech-native architecture selection (incl. hybrid/gated designs) [source]
Real-time voice APIs - OpenAI Realtime (gpt-realtime) & Gemini Live API: transport, sessions, sideband tools [source]
Turn detection & barge-in - VAD, endpointing, semantic/model-integrated detection (Flux), backchannel classification, push-to-talk [source]
Voice latency budgeting & optimization - the <800 ms budget, TTFT dominance, streaming, colocation [source]
Voice component selection - streaming STT/TTS landscape and benchmarks (Deepgram/AssemblyAI/Whisper; ElevenLabs/Cartesia/Gradium/Deepgram) [source]
Voice orchestration frameworks & telephony - LiveKit Agents, Pipecat, TEN; SIP/Twilio/PSTN bridging [source]
Managed voice-agent platforms - Vapi vs. Retell vs. Bland (build-vs-buy, compliance, telephony) [source]
Voice UX design - confidence-tiered confirmation, error recovery, consent, persona, multilingual, accessibility [source]
Voice-agent evaluation - 4-layer framework, WER limits, interruption accuracy, MOS, TSR/FCR, simulated-caller testing [source]

Knowledge Gaps & Low-Confidence Areas

TTS latency benchmark numbers (Section 5) rely on a single vendor-published benchmark (Gradium) - directional, not independently verified. Confidence: Medium-Low. [source]
S2S adoption-percentage projections (<15% H1 2026 → 25–30% H2 2026) come from one analyst source (Coval) - Low confidence; treat as a forecast, not a measured figure. [source]
Vendor latency claims (Retell sub-500 ms, Bland 600–900 ms) are largely from competitor-comparison pages (often Retell's own) - directionally consistent but commercially motivated. Confidence: Medium. [source]
Exact OpenAI/Gemini interruption-handling internals (how barge-in is implemented inside the GA models) are not publicly detailed and shade into model-internal territory owned by multimodal-llm-architecture. [source]
Cost figures vary widely by configuration and date; treat all $/min numbers as order-of-magnitude. [source]

Sources

Coval - Speech-to-Speech vs Cascaded: Which Architecture Should You Deploy? [source]
Speko - Speech-to-Speech vs Cascaded Pipelines [source]
Hamming AI - Are Speech-to-Speech Models Ready to Replace Cascade Models? [source]
Softcery - Real-Time (S2S) vs Turn-Based (Cascading STT/TTS) Architecture [source]
Krzysztof Sopyla - Speech-to-Speech Models in 2026: Three Architectural Bets [source]
OpenAI - Developer notes on the Realtime API [source]
OpenAI - Realtime and audio guide [source]
OpenAI - Realtime API with WebRTC [source]
OpenAI - Realtime API with WebSocket [source]
OpenAI - Webhooks and server-side controls [source]
OpenAI - Voice activity detection (VAD) [source]
webrtcHacks - How OpenAI does WebRTC in the new gpt-realtime [source]
Google Cloud - How to use Gemini Live API Native Audio in Vertex AI [source]
Google AI - Live API capabilities guide [source]
blog.google - Gemini 2.5 Native Audio upgrade [source]
LiveKit - Turn Detection for Voice Agents: VAD, Endpointing, Model-Based Detection [source]
LiveKit - Turns overview (docs) [source]
Microsoft - Configuring Noise Detection and Barge-In with Azure Voice Live API [source]
SkyScribe - AI Voice Recognition: Barge-In, Turn-Taking, and VAD [source]
futureagi - Voice AI Barge-In and Turn-Taking: A 2026 Implementation Guide [source]
Deepgram - Introducing Flux: Conversational Speech Recognition [source]
Deepgram - Getting Started with Flux (docs) [source]
Hamming AI - Voice AI Latency: What's Fast, What's Slow, and How to Fix It [source]
Smallest.ai - Designing Voice Assistants: STT, LLM, TTS, Tools, and Latency Budget [source]
Twilio - Core Latency in AI Voice Agents [source]
Cresta - Engineering for Real-Time Voice Agent Latency [source]
futureagi - Best Speech-to-Text APIs in 2026 [source]
Gradium - TTS Latency Benchmark 2026 (TTFA) [source]
futureagi - ElevenLabs vs Cartesia: 2026 Streaming TTS Comparison [source]
Softcery - How to Choose STT and TTS for Voice Agents [source]
f22labs - LiveKit vs Pipecat Voice AI Platforms [source]
WebRTC.ventures - Choosing a Voice AI Agent Production Framework [source]
LiveKit - Telephony introduction (docs) [source]
Retell AI - Vapi vs Retell comparison [source]
ainora - Retell AI vs Bland AI vs Vapi (2026) [source]
SuperDupr - Vapi vs Bland vs Retell (2026) [source]
InfoWorld - Building enterprise voice AI agents: A UX approach [source]
Clearly Design - Designing for AI Failures: Error States and Recovery [source]
Smashing Magazine - Designing for Agentic AI: Practical UX Patterns [source]
fuselabcreative - Voice UI Design Guide 2026 [source]
dev.to/Kuldeep Paul - How to Evaluate Voice AI Agents: End-to-End Framework [source]
Cekura - A Developer's Guide to Voice AI Evaluation Metrics (2026) [source]
Hamming AI - Voice Agent Evaluation Metrics Guide [source]
Hamming AI - Voice Agent Testing Guide [source]
Braintrust - How to evaluate voice agents [source]
appinventiv - AI Voice Agent Challenges: 8 Failures & How to Fix Them [source]
relinns - 7 Ways to Improve Your AI Voice Agent Latency [source]
Ultravox - Understanding Latency in Voice AI Systems [source]
LiveKit - agents (GitHub) [source]
Cekura - Pipecat vs. LiveKit: The Real Difference [source]

Methodology

Ran 10 web searches and 4 full-page deep-reads (3 fetch timeouts were retried sequentially and succeeded) across the web using the built-in WebSearch/WebFetch fallback (firecrawl/exa MCPs were not configured in this environment; per skill guidance, source-count target was raised ~50% to compensate). Sub-questions investigated: (1) cascaded vs S2S architecture, (2) OpenAI Realtime + Gemini Live APIs, (3) turn detection / VAD / barge-in, (4) latency budgeting, (5) STT/TTS components, (6) orchestration frameworks + telephony, (7) managed platforms, (8) voice UX, (9) evaluation, (10) anti-patterns. Each sub-question is backed by 3+ independent sources except the TTS latency benchmark (single source, flagged Medium-Low) and S2S adoption forecast (single source, flagged Low). Injection guard honored - all fetched content treated as data; no adversarial redirect content encountered. [source]

Children

No children recorded.

← the whole tree · 3D view· how to read this page