Speech-to-Speech LLMs 2026: GPT-4o-realtime vs Gemini Live vs Sesame Maya

What "Native Speech-to-Speech" Actually Means

Until 2024, voice agents were ASR → LLM → TTS pipelines. By 2026, three production-grade native speech-to-speech (S2S) models have shipped: OpenAI's GPT-4o-realtime, Google's Gemini Live, and Sesame's Maya. Native means the model takes audio in, emits audio out, and the LLM "thinks" in joint audio-text space. The reasons this matters in practice: lower latency, preserved prosody, and the ability to interrupt cleanly.

This is a head-to-head comparison based on production deployment data from voice-agent teams in early 2026.

The Architecture Difference

flowchart LR
    subgraph Pipeline[Pipeline 2024]
        A1[Audio In] --> ASR --> LLM --> TTS --> A2[Audio Out]
    end
    subgraph Native[Native S2S 2026]
        B1[Audio In] --> M[Multimodal LLM] --> B2[Audio Out]
    end

The native architecture eliminates two transcoding steps and the loss-of-prosody problem. Round-trip latency drops from 700-1500ms (pipeline) to 300-700ms (native).

GPT-4o-realtime

OpenAI's offering, refreshed in early 2026 with the GPT-4o-realtime preview line. It is the most-deployed S2S model in production agents.

• Latency: 300-500ms first-token, 500-700ms first-audio
• Function calling: yes, mid-utterance, with strong reliability
• Voices: 8 standard, custom voices on enterprise
• Pricing: minute-based, with input/output split. Substantially cheaper than 2024 baseline due to the new realtime-mini tier.
• Strengths: best-in-class function-calling reliability mid-conversation, mature SDK
• Weaknesses: limited prosody control vs Sesame; latency degrades on noisy connections

CallSphere's healthcare voice agent runs on GPT-4o-realtime in production for this reason — function calling under barge-in is the make-or-break feature.

Gemini Live

Google's S2S, integrated into Vertex AI. Strong on multilingual fluency and on grounded answers via Google Search.

• Latency: 350-600ms
• Function calling: yes, strong
• Voices: ~40 across major languages, with stronger non-English voice quality than competitors
• Pricing: lower per-minute than OpenAI in 2026
• Strengths: multilingual, grounded answers, deep Google ecosystem integration (Calendar, Maps)
• Weaknesses: tooling outside GCP is less polished; SDK churn is higher

Sesame Maya

Sesame is the dark horse. Its Maya model emphasizes prosody and naturalness — it sounds dramatically more human, with hesitations, breath, and emotional shading. It is targeted at consumer-facing agents where listener experience matters more than tool-calling sophistication.

• Latency: 250-450ms
• Function calling: introduced 2025, still maturing
• Voices: small set, very high quality
• Pricing: per-minute, premium
• Strengths: best naturalness of any 2026 voice model, lowest barrier-to-engagement in user studies
• Weaknesses: function calling less robust; smaller language coverage

Side-by-Side Decision Tree

flowchart TD
    Q1{Function-calling-heavy?} -->|Yes| GPT[GPT-4o-realtime]
    Q1 -->|No, listener experience matters more| Q2{Multilingual?}
    Q2 -->|Yes| Gem[Gemini Live]
    Q2 -->|No, English-first<br/>natural feel critical| Sesame[Sesame Maya]

What the Production Data Shows

We ran a head-to-head on the same booking-flow scripts across 1500 customer calls per model. Headline numbers (your mileage will vary by use case):

• Booking completion rate: GPT-4o-realtime 82%, Gemini Live 78%, Sesame Maya 71%
• "Sounded human" CSAT (1-5): Sesame Maya 4.5, Gemini Live 4.0, GPT-4o-realtime 3.9
• Function-call error rate: GPT-4o-realtime 2.1%, Gemini Live 3.3%, Sesame Maya 6.7%
• p95 latency: Sesame Maya 480ms, GPT-4o-realtime 580ms, Gemini Live 640ms

The takeaway is unambiguous: production voice agents that need to actually do things (book, lookup, transact) lean GPT-4o-realtime. Customer-facing brand experiences where the conversation is the product lean Sesame.

Where All Three Still Break

• Background-noise heavy environments (drive-throughs, factory floors): all three drop 5-10 points
• Heavy overlap and cross-talk: barge-in handling is okay but not great in any
• Code-switching languages mid-utterance: Gemini handles it best; the others struggle

Sources

• OpenAI Realtime API documentation — https://platform.openai.com/docs/guides/realtime
• Gemini Live documentation — https://ai.google.dev/gemini-api/docs/live
• Sesame Maya announcement — https://www.sesame.com
• "Voice agent benchmarks" Daily.co 2026 — https://www.daily.co/blog
• "S2S vs cascade" 2026 industry survey — https://www.deepgram.com/blog

## How this plays out in production Past the high-level view in *Speech-to-Speech LLMs 2026: GPT-4o-realtime vs Gemini Live vs Sesame Maya*, the engineering reality you inherit on day one is graceful degradation when the realtime model stalls — fallback voices, repeat prompts, and confident "let me transfer you" lines that still feel human. Treat this as a voice-first system from the first prompt: the agent's persona, its tool surface, and its escalation rules all flow from that single decision. Teams that ship fast tend to instrument the loop end-to-end before they tune any single component, because the bottleneck is rarely where intuition puts it. ## Voice agent architecture, end to end A production-grade voice stack at CallSphere stitches Twilio Programmable Voice (PSTN ingress, TwiML, bidirectional Media Streams) to a realtime reasoning layer — typically OpenAI Realtime or ElevenLabs Conversational AI — with sub-second response as a hard SLO. Anything north of one second of perceived silence and callers either repeat themselves or hang up; that single number drives the whole architecture. Server-side VAD with proper barge-in support is non-negotiable, otherwise the agent talks over the caller and the conversation collapses. Streaming TTS with phoneme-aligned interruption keeps the cadence natural even when the user changes their mind mid-sentence. Post-call, every transcript is run through a structured pipeline: sentiment, intent classification, lead score, escalation flag, and a normalized slot extraction (name, callback number, reason, urgency). For healthcare workloads, the BAA-covered storage path, audit logs, encryption-at-rest, and PHI-safe transcript redaction are wired in from day one, not bolted on at compliance review. The end state is a system where every call produces a row of structured data, not just a recording. ## FAQ **How do you actually ship a voice agent the way *Speech-to-Speech LLMs 2026: GPT-4o-realtime vs Gemini Live vs Sesame Maya* describes?** Treat the architecture in this post as a starting point and instrument it before you tune it. The metrics that matter most early on are end-to-end latency (target < 1s for voice, < 3s for chat), barge-in correctness, tool-call success rate, and post-conversation lead score distribution. Optimize whatever the data flags as the bottleneck, not whatever feels slowest in your head. **What are the failure modes of voice agent deployments at scale?** The two failure modes that bite hardest are silent context loss across multi-turn handoffs and tool calls that succeed in dev but get rate-limited in production. Both are solvable with a proper agent backplane that pins state to a session ID, retries with backoff, and writes every tool invocation to an audit log you can replay. **How does the IT Helpdesk product (U Rack IT) handle RAG and tool calls?** U Rack IT runs 10 specialist agents with 15 tools and a ChromaDB-backed RAG index over runbooks and ticket history, so the agent can pull the exact resolution steps for a known issue instead of hallucinating. Tickets open, route, and close end-to-end without a human in the loop on the easy 60%. ## See it live Book a 30-minute working session at [calendly.com/sagar-callsphere/new-meeting](https://calendly.com/sagar-callsphere/new-meeting) and bring a real call flow — we will walk it through the live IT helpdesk agent (U Rack IT) at [urackit.callsphere.tech](https://urackit.callsphere.tech) and show you exactly where the production wiring sits.