Voice and Real-Time Agent Design

Parent: AI Agent Ecosystems · Published reference · snapshot 2026-09-08 · skill ai-agent-engineering

Also known as: Voice & Real-Time Agent Design (2024–2026)

> Scope note: This report covers the application / design layer of voice agents — architecture choices, real-time APIs, turn-taking, latency engineering, orchestration frameworks, components, UX, and

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Voice & Real-Time Agent Design (2024–2026): Research Report

> Scope note: This report covers the application / design layer of voice agents - architecture choices, real-time APIs, turn-taking, latency engineering, orchestration frameworks, components, UX, and evaluation. Model-internal speech architecture (Whisper encoders, the Thinker–Talker decomposition, audio tokenization) is deliberately out of scope and is covered by the existing multimodal-llm-architecture reference. Cross-reference that reference for "how the model produces audio"; this report is "how you build and operate a voice product around such models." [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#voice-real-time-agent-design-20242026-research-report]

Overview

A voice agent is a real-time conversational system that listens, reasons, and speaks back inside a turn-taking loop tight enough to feel human. The dominant 2024–2026 design tension is cascaded pipelines (STT → LLM → TTS as discrete, swappable stages) versus speech-native / speech-to-speech (S2S) models (one model that ingests and emits audio directly, e.g., OpenAI's gpt-realtime and Google's Gemini Live API). Cascaded wins on control, debuggability, compliance, and provider choice; S2S wins on latency and emotional prosody but is hard to audit and gate (Coval, Speko, Hamming). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#overview]
As of mid-2026, most production voice agents still use cascaded architectures, with S2S adoption projected below 15% in H1 2026 and rising to roughly 25–30% by end of H2 2026 as evaluation tooling and compliance frameworks mature (Coval). The engineering north star is a voice-to-voice latency budget under 800 ms, and the field has converged on streaming everything, semantic turn detection, and first-class barge-in as the table-stakes pattern (Hamming, Smallest.ai). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#overview]

1. Architecture: Cascaded vs. Speech-Native (S2S)

Cascaded = an orchestration layer coordinating specialized services: VAD → streaming STT → LLM → streaming TTS (Softcery). S2S = a single model that "processes the entire exchange in a single latent space with no intermediate representations" (Speko). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#1-architecture-cascaded-vs-speech-native-s2s]
Trade-off summary (confidence: High - multiple independent sources agree): [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#1-architecture-cascaded-vs-speech-native-s2s]
When to use which: S2S for premium/empathetic experiences (mental-health, coaching, luxury, multilingual); cascaded for high-volume tier-1 support, regulated industries, and complex tool-driven workflows (Coval). A streaming cascade (Deepgram + GPT-4o-mini + Cartesia) can already hit sub-1-second latency, narrowing S2S's main advantage (Speko). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#1-architecture-cascaded-vs-speech-native-s2s]

2. Real-Time APIs

OpenAI Realtime API graduated to GA in 2025 with the gpt-realtime S2S model (GA snapshot gpt-realtime-2025-08-28, 32,768-token context, 4,096 max output) (OpenAI Realtime blog, OpenAI docs). Key design facts: [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#2-real-time-apis]
- Transport: WebRTC is recommended for client media (browser/mobile) for "more consistent performance"; WebSockets for server media pipelines (phone calls, broadcast ingest) (OpenAI docs, webrtcHacks). WebRTC sessions use the new https://api.openai.com/v1/realtime/calls endpoint. [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#2-real-time-apis]
- Auth: Ephemeral client secrets via POST /v1/realtime/client_secrets keep permanent keys off the client; the GA interface drops the OpenAI-Beta: realtime=v1 header (OpenAI Realtime blog). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#2-real-time-apis]
- Sessions: Configured via session.update (instructions, turn-detection mode, truncation) (OpenAI Realtime blog). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#2-real-time-apis]
- Tool use / function calling: Keep tools and business logic server-side via a sideband control channel - two connections to one session (one client, one server); the server monitors, updates instructions, and answers tool calls. GA adds asynchronous function calling with automatic placeholder responses ("I'm still waiting on that") to suppress hallucination during pending calls (OpenAI Realtime blog, OpenAI server-controls docs). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#2-real-time-apis]
Gemini Live API (GA on Vertex AI) is built on Gemini 2.5 Flash Native Audio - raw audio through a single low-latency model, with 30 HD voices in 24 languages, emotion-aware responses, live speech-to-speech translation, and tool use (function calling + Google Search) (Google Cloud blog, Gemini Live API docs, blog.google). It is multimodal - agents can converse about live visual streams (charts, video) alongside spoken input (Google Cloud blog). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#2-real-time-apis]

3. Turn Detection, Endpointing, and Barge-In

The hardest UX problem in voice agents is knowing when the user is done speaking and letting them interrupt. Three layers, increasing in sophistication (LiveKit, LiveKit docs): [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#3-turn-detection-endpointing-and-barge-in]
- VAD (server VAD): Classifies each audio frame as speech/non-speech in real time (Silero VAD is the widely-used reference model). Cheap but dumb - it only sees energy, not meaning (LiveKit). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#3-turn-detection-endpointing-and-barge-in]
- Endpointing: Uses a silence-timeout on top of VAD to declare end-of-turn. The core latency trap: "A silence timeout set to 800 ms adds nearly a full second to every single response before the pipeline even starts" (LiveKit). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#3-turn-detection-endpointing-and-barge-in]
- Semantic / model-based turn detection: Reads the partial transcript and infers whether sentence structure is complete. "Can trigger before trailing silence occurs, which is the main latency advantage," letting silence thresholds drop to 200–300 ms without raising false interruptions (LiveKit, appinventiv). OpenAI exposes both Server VAD and Semantic VAD modes (OpenAI VAD docs); Azure offers a noise-resilient Semantic VAD (Microsoft). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#3-turn-detection-endpointing-and-barge-in]
Two persistent VAD-only failure modes: false positives (pauses/hesitations read as turn ends) and delayed responses (strict silence thresholds stall replies after the user is semantically done) (LiveKit, SkyScribe). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#3-turn-detection-endpointing-and-barge-in]
Model-integrated turn detection is the 2026 frontier: Deepgram Flux is a "conversational speech recognition" model with first-of-its-kind model-integrated end-of-turn detection, detecting end-of-turns in ~260 ms (median <300 ms, p95 ~1.5 s) at Nova-3 accuracy - collapsing ASR + VAD + endpointing into one model (Deepgram Flux intro, Deepgram Flux docs). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#3-turn-detection-endpointing-and-barge-in]
Barge-in (interruption): Keep the turn-detection layer active even while the agent is speaking; on detected user speech, cancel the current TTS stream and hand control back to STT immediately. Echo cancellation runs client-side; push-to-talk is the fallback for devices lacking echo cancellation (LiveKit, futureagi). Backchanneling ("mm-hm", "yeah") is the subtle case: pure VAD misreads backchannels as either silence or a full barge-in, so the 2026 stack moves to dedicated turn-taking models that classify backchannel vs. barge-in as a learned signal (appinventiv, DEV/Mishra). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#3-turn-detection-endpointing-and-barge-in]

4. Latency Engineering

Target: under 800 ms end-to-end for production voice agents (Hamming, Ultravox). Component latencies are cumulative and sequential - a naïve "each stage waits for the previous" pipeline easily exceeds 2–3 s (Smallest.ai). Representative budget (confidence: High - consistent across Hamming, Smallest.ai, Twilio): [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#4-latency-engineering]
Sources: Hamming, Smallest.ai. [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#4-latency-engineering]
Where it accrues: LLM inference is the dominant cost - Hamming attributes ~70% of total latency to it, "making model selection critical" (Hamming). For voice, TTFT matters more than total generation time, because TTS can begin streaming as soon as the first sentence is complete (Smallest.ai, Cresta). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#4-latency-engineering]
Mitigations: streaming STT (save 100–200 ms), streaming TTS (save 200–400 ms on TTFB), fast LLMs (GPT-4o-mini ~400 ms TTFT, Claude 3.5 Haiku ~360 ms), semantic endpointing (save 200–400 ms), and geographic colocation (US-coast-to-coast +60–80 ms, US–Europe +80–150 ms, US–Asia +150–250 ms) (Hamming, relinns). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#4-latency-engineering]

5. Components: Streaming ASR/STT and TTS

STT (May 2026 landscape): Deepgram leads on voice-agent latency and end-of-speech detection (Nova-3 hosted WER ~5.26% on its own real-world test set); Nova-3 + Flux is positioned as the May-2026 voice-agent default. ElevenLabs Scribe v2 Realtime leads multilingual real-time; AssemblyAI leads transcript intelligence; Whisper covers 57+ languages but is beaten by Scribe v2 on its 30 (futureagi STT, Softcery STT/TTS, AssemblyAI). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#5-components-streaming-asrstt-and-tts]
TTS time-to-first-audio (TTFA) benchmark, May 2026 (confidence: Medium - single benchmark source, vendor-published): Gradium TTS 155 ms P50 (2 ms IQR); Cartesia Sonic-3 188 ms P50 but 100 ms IQR (50× wider); ElevenLabs Turbo v2.5 264 ms P50 (28 ms IQR), Flash v2.5 288 ms; Deepgram Aura-2 313 ms (Gradium benchmark). Positioning consensus: ElevenLabs = voice realism / cloning benchmark; Cartesia = streaming-latency benchmark (futureagi TTS, Cartesia, Cekura TTS). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#5-components-streaming-asrstt-and-tts]

6. Voice UX / Design

Voice UI's defining constraint: nothing is visible and the spoken sentence is gone the moment it lands, so error recovery is a primary discipline, not an edge case (InfoWorld, fuselabcreative). Key patterns: [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#6-voice-ux-design]
- Confidence-tiered confirmation: High confidence → act + implicit confirmation ("I've sent the invoice to your inbox"); medium confidence → clarify ("I found 3 contacts named John - which one?"); low confidence → graceful fallback (InfoWorld). Implicit confirmation beats explicit yes/no questions in enterprise flows. [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#6-voice-ux-design]
- Error recovery as trust: users forgive the first error; doubt by the second; "doesn't work" by the third - so agents must state when confused and offer concrete next steps, not cryptic failures (InfoWorld, Clearly Design). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#6-voice-ux-design]
- Consent before consequential action: the foundational moment before an agent takes a significant action (Smashing). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#6-voice-ux-design]

7. Evaluation

A 4-layer voice-agent quality framework: (1) Infrastructure - audio quality, latency, ASR/TTS perf; (2) Execution - intent classification, response accuracy, tool-calling logic; (3) User-behavior - interruption handling, conversation flow, sentiment; (4) Business-outcome - containment rate, first-call resolution (FCR), escalation (dev.to/Paul, Cekura metrics, Hamming metrics). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#7-evaluation]
Headline metrics and targets: WER = (S+D+I)/total words, target <5% for enterprise - but WER "ignores interaction dynamics" (barge-in, endpointing, turn-taking) and underestimates real-world UX under noise/accents/far-field (dev.to/Paul). Interruption handling: agent should stop within 200 ms of user speech and acknowledge/address the interruption >90% of the time (dev.to/Paul). MOS (naturalness) ~4.5/5 for near-human; Task Success Rate / FCR target ~85%+ (dev.to/Paul, Hamming testing, Braintrust). Testing tooling (Hamming, Coval, Cekura, Braintrust) emphasizes simulated callers / regression / load testing before live traffic, and per-component instrumentation tagged by call_id/turn_id (Hamming testing, Softcery QA). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#7-evaluation]

Tools / Frameworks

Orchestration frameworks (open source): [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#tools-frameworks]
- Pipecat - Python, pipeline/frame model stringing VAD → STT → LLM → TTS; "most elegant" for 1:1 voice assistants; transport-agnostic (WebSocket, WebRTC via Daily, Twilio media streams) but you assemble production deployment yourself (f22labs, Cekura framework, AssemblyAI frameworks). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#tools-frameworks]
- LiveKit Agents - WebRTC-first; ships production transport (rooms, participant mgmt, track routing, egress/ingress, recording, SFU) in Go/Python/Node; best for multi-participant and latency-sensitive deployments that also need SIP (Mansoori, LiveKit GitHub, WebRTC.ventures). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#tools-frameworks]
- TEN - third open-source contender alongside LiveKit/Pipecat (Medium/Garcia). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#tools-frameworks]
Managed platforms (build-faster, telephony-included): [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#tools-frameworks]
- Vapi - middleware/BYO-everything (your LLM, TTS, STT, telephony Twilio/Vonage/Telnyx); ~500–800 ms latency; weaker native telephony (no warm transfer / branded calls / native SIP) and gated compliance (Retell vs Vapi, SuperDupr). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#tools-frameworks]
- Retell AI - managed, voice-quality + sub-500 ms latency leader; ships warm transfer, branded calls, native SIP trunking, KB retrieval, HIPAA/SOC2/GDPR on every plan (Retell vs Vapi, ainora). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#tools-frameworks]
- Bland AI - API-first, high-volume outbound; ~600–900 ms latency, all-inclusive pricing, strong data governance (SuperDupr, ainora). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#tools-frameworks]
Telephony integration: PSTN reaches a WebRTC room via SIP trunking - buy a number (Twilio/Vonage/Telnyx), point its Voice URL at the framework's SIP URI. LiveKit telephony supports DTMF, call transfers, secure trunking, HD voice, region pinning, noise cancellation, plus connectors that bridge Twilio/WhatsApp without manual SIP config (LiveKit telephony docs, LiveKit agents-integration). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#tools-frameworks]

Practical Patterns

Stream every stage; never block. Begin TTS on the first complete sentence of the LLM stream; begin LLM on the first finalized STT segment. Naïve sequential pipelines hit 2–3 s (Smallest.ai, Hamming). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#practical-patterns]
Optimize for LLM TTFT, not total tokens. It's ~70% of the budget; pick a fast tier model and keep the prompt small (Hamming). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#practical-patterns]
Use semantic (or model-integrated) turn detection to push silence thresholds to 200–300 ms without false cut-offs; Deepgram Flux folds end-of-turn into ASR (~260 ms) (LiveKit, Deepgram Flux). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#practical-patterns]
Make barge-in first-class. Keep turn detection live during playback; cancel TTS within 200 ms; distinguish backchannel from interruption with a turn-taking model (LiveKit, appinventiv). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#practical-patterns]
Keep tools/business logic server-side via a sideband channel (OpenAI Realtime) so secrets and policy never touch the client (OpenAI Realtime blog). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#practical-patterns]
Match transport to context: WebRTC for browser/mobile clients; WebSocket for server-side / telephony media (OpenAI docs). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#practical-patterns]
Tier confirmations by confidence and treat error recovery as a primary design surface (InfoWorld). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#practical-patterns]
Instrument per component (call_id/turn_id), watch p95 tail latency, and gate releases with simulated-caller regression tests (Hamming testing, appinventiv). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#practical-patterns]
Default cascaded for regulated/tool-heavy; reserve S2S for empathy-first experiences, or use a hybrid (S2S for conversation, cascade for compliance-gated branches) (Coval). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#practical-patterns]

Anti-Patterns

Blocking pipeline - waiting for the full LLM response, then starting TTS, then playing audio; adds hundreds of ms per turn for no reason (appinventiv). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#anti-patterns]
Over-long silence timeout - an 800 ms endpoint timeout adds ~1 s to every turn (LiveKit). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#anti-patterns]
Endpointing that holds too long - semantic detection waiting past the fallback silence timeout shows up as a discrete tail-latency jump, not a gradual rise (appinventiv). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#anti-patterns]
No / poor barge-in - speaking over the user or ignoring interruptions; treating backchannels as full barge-ins (futureagi, appinventiv). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#anti-patterns]
Retry cascades without hard timeouts - a 30 s API outlier strands the caller even if it averages away in dashboards (appinventiv). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#anti-patterns]
Weak multi-turn context - failing to resolve "cancel that one" three turns deep; demos pass, production fails (appinventiv). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#anti-patterns]
Deploying S2S into regulated flows - no text intermediary means non-compliant audio can reach the customer before any filter (Coval). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#anti-patterns]
WER-only evaluation - ignores barge-in, endpointing, turn-taking, and real-world noise/accents (dev.to/Paul). [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#anti-patterns]

Suggested Sub-Concepts (future child concepts)

Cascaded vs. speech-native architecture selection (incl. hybrid/gated designs) [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#suggested-sub-concepts-future-child-concepts]
Real-time voice APIs - OpenAI Realtime (gpt-realtime) & Gemini Live API: transport, sessions, sideband tools [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#suggested-sub-concepts-future-child-concepts]
Turn detection & barge-in - VAD, endpointing, semantic/model-integrated detection (Flux), backchannel classification, push-to-talk [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#suggested-sub-concepts-future-child-concepts]
Voice latency budgeting & optimization - the <800 ms budget, TTFT dominance, streaming, colocation [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#suggested-sub-concepts-future-child-concepts]
Voice component selection - streaming STT/TTS landscape and benchmarks (Deepgram/AssemblyAI/Whisper; ElevenLabs/Cartesia/Gradium/Deepgram) [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#suggested-sub-concepts-future-child-concepts]
Voice orchestration frameworks & telephony - LiveKit Agents, Pipecat, TEN; SIP/Twilio/PSTN bridging [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#suggested-sub-concepts-future-child-concepts]
Managed voice-agent platforms - Vapi vs. Retell vs. Bland (build-vs-buy, compliance, telephony) [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#suggested-sub-concepts-future-child-concepts]
Voice UX design - confidence-tiered confirmation, error recovery, consent, persona, multilingual, accessibility [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#suggested-sub-concepts-future-child-concepts]
Voice-agent evaluation - 4-layer framework, WER limits, interruption accuracy, MOS, TSR/FCR, simulated-caller testing [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#suggested-sub-concepts-future-child-concepts]

Knowledge Gaps & Low-Confidence Areas

TTS latency benchmark numbers (Section 5) rely on a single vendor-published benchmark (Gradium) - directional, not independently verified. Confidence: Medium-Low. [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#knowledge-gaps-low-confidence-areas]
S2S adoption-percentage projections (<15% H1 2026 → 25–30% H2 2026) come from one analyst source (Coval) - Low confidence; treat as a forecast, not a measured figure. [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#knowledge-gaps-low-confidence-areas]
Vendor latency claims (Retell sub-500 ms, Bland 600–900 ms) are largely from competitor-comparison pages (often Retell's own) - directionally consistent but commercially motivated. Confidence: Medium. [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#knowledge-gaps-low-confidence-areas]
Exact OpenAI/Gemini interruption-handling internals (how barge-in is implemented inside the GA models) are not publicly detailed and shade into model-internal territory owned by multimodal-llm-architecture. [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#knowledge-gaps-low-confidence-areas]
Cost figures vary widely by configuration and date; treat all $/min numbers as order-of-magnitude. [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#knowledge-gaps-low-confidence-areas]

Sources

Coval - Speech-to-Speech vs Cascaded: Which Architecture Should You Deploy? [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Speko - Speech-to-Speech vs Cascaded Pipelines [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Hamming AI - Are Speech-to-Speech Models Ready to Replace Cascade Models? [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Softcery - Real-Time (S2S) vs Turn-Based (Cascading STT/TTS) Architecture [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Krzysztof Sopyla - Speech-to-Speech Models in 2026: Three Architectural Bets [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
OpenAI - Developer notes on the Realtime API [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
OpenAI - Realtime and audio guide [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
OpenAI - Realtime API with WebRTC [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
OpenAI - Realtime API with WebSocket [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
OpenAI - Webhooks and server-side controls [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
OpenAI - Voice activity detection (VAD) [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
webrtcHacks - How OpenAI does WebRTC in the new gpt-realtime [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Google Cloud - How to use Gemini Live API Native Audio in Vertex AI [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Google AI - Live API capabilities guide [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
blog.google - Gemini 2.5 Native Audio upgrade [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
LiveKit - Turn Detection for Voice Agents: VAD, Endpointing, Model-Based Detection [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
LiveKit - Turns overview (docs) [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Microsoft - Configuring Noise Detection and Barge-In with Azure Voice Live API [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
SkyScribe - AI Voice Recognition: Barge-In, Turn-Taking, and VAD [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
futureagi - Voice AI Barge-In and Turn-Taking: A 2026 Implementation Guide [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Deepgram - Introducing Flux: Conversational Speech Recognition [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Deepgram - Getting Started with Flux (docs) [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Hamming AI - Voice AI Latency: What's Fast, What's Slow, and How to Fix It [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Smallest.ai - Designing Voice Assistants: STT, LLM, TTS, Tools, and Latency Budget [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Twilio - Core Latency in AI Voice Agents [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Cresta - Engineering for Real-Time Voice Agent Latency [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
futureagi - Best Speech-to-Text APIs in 2026 [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Gradium - TTS Latency Benchmark 2026 (TTFA) [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
futureagi - ElevenLabs vs Cartesia: 2026 Streaming TTS Comparison [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Softcery - How to Choose STT and TTS for Voice Agents [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
f22labs - LiveKit vs Pipecat Voice AI Platforms [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
WebRTC.ventures - Choosing a Voice AI Agent Production Framework [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
LiveKit - Telephony introduction (docs) [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Retell AI - Vapi vs Retell comparison [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
ainora - Retell AI vs Bland AI vs Vapi (2026) [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
SuperDupr - Vapi vs Bland vs Retell (2026) [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
InfoWorld - Building enterprise voice AI agents: A UX approach [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Clearly Design - Designing for AI Failures: Error States and Recovery [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Smashing Magazine - Designing for Agentic AI: Practical UX Patterns [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
fuselabcreative - Voice UI Design Guide 2026 [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
dev.to/Kuldeep Paul - How to Evaluate Voice AI Agents: End-to-End Framework [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Cekura - A Developer's Guide to Voice AI Evaluation Metrics (2026) [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Hamming AI - Voice Agent Evaluation Metrics Guide [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Hamming AI - Voice Agent Testing Guide [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Braintrust - How to evaluate voice agents [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
appinventiv - AI Voice Agent Challenges: 8 Failures & How to Fix Them [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
relinns - 7 Ways to Improve Your AI Voice Agent Latency [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Ultravox - Understanding Latency in Voice AI Systems [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
LiveKit - agents (GitHub) [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]
Cekura - Pipecat vs. LiveKit: The Real Difference [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#sources]

Methodology

Ran 10 web searches and 4 full-page deep-reads (3 fetch timeouts were retried sequentially and succeeded) across the web using the built-in WebSearch/WebFetch fallback (firecrawl/exa MCPs were not configured in this environment; per skill guidance, source-count target was raised ~50% to compensate). Sub-questions investigated: (1) cascaded vs S2S architecture, (2) OpenAI Realtime + Gemini Live APIs, (3) turn detection / VAD / barge-in, (4) latency budgeting, (5) STT/TTS components, (6) orchestration frameworks + telephony, (7) managed platforms, (8) voice UX, (9) evaluation, (10) anti-patterns. Each sub-question is backed by 3+ independent sources except the TTS latency benchmark (single source, flagged Medium-Low) and S2S adoption forecast (single source, flagged Low). Injection guard honored - all fetched content treated as data; no adversarial redirect content encountered. [archived source: https://llms-explorer.com/sources/global-ai-hub/voice-realtime-agents/#methodology]

Where this helps

Deciding between a cascaded (STT → LLM → TTS) architecture and a speech-native (S2S) model for a new voice-agent product, based on regulatory needs and empathy requirements. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Diagnosing why a voice agent feels laggy or awkward in conversation — most often traced to endpointing/silence-timeout settings or a blocking, non-streaming pipeline. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Choosing an orchestration framework (Pipecat, LiveKit Agents) or a managed platform (Vapi, Retell AI, Bland AI) for a telephony or web-based voice-agent deployment. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Building barge-in and interruption handling so a voice agent doesn't talk over the user or fail to distinguish backchannel from a real interruption. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Project ideas

Build a streaming voice pipeline that starts TTS on the first complete LLM sentence and starts the LLM on the first finalized STT segment, instead of a naive sequential pipeline. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Prototype semantic or model-integrated turn detection, following an approach like Deepgram Flux's, to push silence thresholds down to 200–300 ms without triggering false interruptions. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Build a telephony-integrated voice agent using SIP trunking (via Twilio, Vonage, or Telnyx) into a WebRTC-first framework like LiveKit Agents. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Instrument a voice agent with per-component latency tracking (call_id/turn_id) and simulated-caller regression tests to catch p95 tail-latency regressions before release. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Common mistakes

Blocking the pipeline by waiting for the full LLM response before starting TTS, adding hundreds of milliseconds per turn for no benefit. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Setting an overly long silence timeout, such as 800 ms, for endpointing, which adds nearly a full second of latency to every response. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Deploying a speech-to-speech (S2S) model into a regulated flow with no text intermediary, so non-compliant audio can reach the customer before any filter runs. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Evaluating a voice agent on Word Error Rate alone, ignoring barge-in quality, endpointing, turn-taking, and real-world noise or accent robustness. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Known issues

As of mid-2026, S2S adoption is still a minority of production deployments because evaluation tooling and compliance frameworks for it are still maturing relative to cascaded architectures. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
VAD-only turn detection has two persistent failure modes — false positives on pauses/hesitations and delayed responses from strict silence thresholds — that semantic detection reduces but doesn't fully eliminate. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
LLM inference dominates voice-agent latency, so latency engineering is bottlenecked on model choice and time-to-first-token more than on the STT/TTS components most teams focus on first. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Managed platforms trade control for speed: a BYO-everything platform can carry weaker native telephony features (no warm transfer, branded calls, or native SIP) compared to a more integrated but less flexible managed offering. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.

Children

Cascaded vs speech-to-speech architecture (frontier)
Real-time APIs (OpenAI Realtime gpt-realtime, Gemini Live) (frontier)
Turn detection & barge-in (VAD, semantic, Deepgram Flux, backchannel) (frontier)
Voice latency budgeting (<800ms, LLM TTFT ~70%) (frontier)
Streaming STT/TTS components (Deepgram, ElevenLabs, Cartesia) (frontier)
Orchestration & telephony (LiveKit, Pipecat, Vapi/Retell/Bland, SIP) (frontier)
Voice UX (confidence-tiered confirmation, error recovery, consent) (frontier)
Voice-agent evaluation (4-layer, WER limits, interruption accuracy, MOS, TSR/FCR) (frontier)

Frontier under this node: Cascaded vs speech-to-speech architecture, Orchestration & telephony (LiveKit, Pipecat, Vapi/Retell/Bland, SIP), Real-time APIs (OpenAI Realtime gpt-realtime, Gemini Live), Streaming STT/TTS components (Deepgram, ElevenLabs, Cartesia), Turn detection & barge-in (VAD, semantic, Deepgram Flux, backchannel), Voice UX (confidence-tiered confirmation, error recovery, consent), Voice latency budgeting (<800ms, LLM TTFT ~70%), Voice-agent evaluation (4-layer, WER limits, interruption accuracy, MOS, TSR/FCR)

← the whole tree · 3D view· how to read this page