How AI Voice Agents Actually Work: Technical Deep Dive (2026 Edition)

The Problem Nobody Warns You About

The first time you build a voice agent that actually works, you notice something strange: the model is smart, the transcription is correct, the voice sounds great — and yet the conversation feels broken. The caller says "hello" and waits two full seconds. They interrupt and the agent keeps talking over them. They ask a question and the agent hallucinates a policy that doesn't exist in your knowledge base.

None of those problems are language model problems. They are systems problems. Voice agents are a distributed, soft-real-time pipeline where every component — microphone capture, VAD, STT, LLM, tool execution, TTS, speaker playback — has to hit a latency budget measured in milliseconds, and has to fail gracefully when any stage misbehaves.

Here is the shape of the pipeline most teams miss when they read "just use the Realtime API":

caller mic
   ↓ (PCM16 @ 24kHz)
carrier / WebRTC bridge
   ↓
server VAD  →  interruption signal
   ↓
STT (streaming)
   ↓ (partial transcripts)
LLM reasoning + tool calls
   ↓ (token stream)
TTS (streaming)
   ↓ (audio frames)
speaker

This post is a full technical walkthrough of how modern AI voice agents work in 2026. It is based on the architecture CallSphere runs in production across healthcare, real estate, salon, after-hours escalation, IT helpdesk, and sales verticals — all of which handle live phone traffic today.

Architecture overview

┌─────────────────────────────────────────────────────────────┐
│                      Caller (PSTN / WebRTC)                 │
└─────────────────────────────────────────────────────────────┘
                │ G.711 ulaw / Opus
                ▼
┌─────────────────────────────────────────────────────────────┐
│  Twilio Media Streams  ←→  Edge bridge (FastAPI WebSocket)  │
└─────────────────────────────────────────────────────────────┘
                │ PCM16 @ 24kHz
                ▼
┌─────────────────────────────────────────────────────────────┐
│  OpenAI Realtime API (gpt-4o-realtime-preview-2025-06-03)   │
│  • Server VAD          • Streaming STT                      │
│  • Function calling    • Streaming TTS                      │
└─────────────────────────────────────────────────────────────┘
                │ tool calls + audio frames
                ▼
┌─────────────────────────────────────────────────────────────┐
│  Tool layer: calendar, CRM, DB, payments, handoff           │
│  Observability: OpenTelemetry spans per stage               │
│  Post-call: GPT-4o-mini summary + sentiment + lead score    │
└─────────────────────────────────────────────────────────────┘

Prerequisites

• Working knowledge of WebSockets and async Python or Node.js.
• An OpenAI account with Realtime API access.
• A Twilio account (or any SIP provider that supports Media Streams / bidirectional audio).
• Familiarity with audio formats: PCM16, sample rates, and G.711 ulaw.
• A Postgres database for session state and call logs.
• Comfort with OpenTelemetry or an equivalent tracing backend.

Step-by-step walkthrough

1. Capture audio at the edge

Your edge service receives audio frames over a WebSocket from the carrier and must forward them to the model without blocking. Back-pressure matters: if you buffer too much, latency explodes; if you buffer too little, you clip the caller.

flowchart LR
    CALLER(["Caller"])
    subgraph TEL["Telephony"]
        SIP["Twilio SIP and PSTN"]
    end
    subgraph BRAIN["Business AI Agent"]
        STT["Streaming STT<br/>Deepgram or Whisper"]
        NLU{"Intent and<br/>Entity Extraction"}
        TOOLS["Tool Calls"]
        TTS["Streaming TTS<br/>ElevenLabs or Rime"]
    end
    subgraph DATA["Live Data Plane"]
        CRM[("CRM and Notes")]
        CAL[("Calendar and<br/>Schedule")]
        KB[("Knowledge Base<br/>and Policies")]
    end
    subgraph OUT["Outcomes"]
        O1(["Booking captured"])
        O2(["CRM record created"])
        O3(["Human handoff"])
    end
    CALLER --> SIP --> STT --> NLU
    NLU -->|Lookup| TOOLS
    TOOLS <--> CRM
    TOOLS <--> CAL
    TOOLS <--> KB
    NLU --> TTS --> SIP --> CALLER
    NLU -->|Resolved| O1
    NLU -->|Schedule| O2
    NLU -->|Escalate| O3
    style CALLER fill:#f1f5f9,stroke:#64748b,color:#0f172a
    style NLU fill:#4f46e5,stroke:#4338ca,color:#fff
    style O1 fill:#059669,stroke:#047857,color:#fff
    style O2 fill:#0ea5e9,stroke:#0369a1,color:#fff
    style O3 fill:#f59e0b,stroke:#d97706,color:#1f2937

from fastapi import FastAPI, WebSocket
import asyncio, base64, json, websockets

app = FastAPI()

OPENAI_WS = "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2025-06-03"

@app.websocket("/twilio/stream")
async def twilio_stream(ws: WebSocket):
    await ws.accept()
    async with websockets.connect(
        OPENAI_WS,
        extra_headers={
            "Authorization": f"Bearer {OPENAI_API_KEY}",
            "OpenAI-Beta": "realtime=v1",
        },
    ) as oai:
        await oai.send(json.dumps({
            "type": "session.update",
            "session": {
                "voice": "alloy",
                "input_audio_format": "pcm16",
                "output_audio_format": "pcm16",
                "turn_detection": {"type": "server_vad", "silence_duration_ms": 400},
                "instructions": "You are a concise, friendly receptionist.",
            },
        }))

        async def from_twilio():
            async for msg in ws.iter_text():
                data = json.loads(msg)
                if data.get("event") == "media":
                    pcm = ulaw_to_pcm16(base64.b64decode(data["media"]["payload"]))
                    await oai.send(json.dumps({
                        "type": "input_audio_buffer.append",
                        "audio": base64.b64encode(pcm).decode(),
                    }))

        async def from_openai():
            async for msg in oai:
                evt = json.loads(msg)
                if evt["type"] == "response.audio.delta":
                    await ws.send_text(json.dumps({
                        "event": "media",
                        "media": {"payload": pcm16_to_ulaw_b64(evt["delta"])},
                    }))

        await asyncio.gather(from_twilio(), from_openai())

2. Let the model handle VAD and interruptions

Server-side VAD is the difference between a conversation and a monologue. When the caller starts speaking while the agent is mid-sentence, the Realtime API fires input_audio_buffer.speech_started — your edge must immediately stop the downstream audio playback so the caller is not talked over.

if evt["type"] == "input_audio_buffer.speech_started":
    await ws.send_text(json.dumps({"event": "clear"}))
    await oai.send(json.dumps({"type": "response.cancel"}))

3. Wire up tool calls

The LLM is only as useful as the tools you give it. Define a small, strongly-typed tool schema, keep the arguments minimal, and validate the output on the server before returning it to the model.

TOOLS = [{
    "type": "function",
    "name": "book_appointment",
    "description": "Book a medical appointment for a patient.",
    "parameters": {
        "type": "object",
        "properties": {
            "patient_id": {"type": "string"},
            "provider_id": {"type": "string"},
            "start_iso": {"type": "string", "description": "ISO 8601 start time"},
            "reason": {"type": "string"},
        },
        "required": ["patient_id", "provider_id", "start_iso"],
    },
}]

4. Stream TTS back to the caller

The Realtime API emits response.audio.delta events as the model speaks. You forward each frame to the carrier without waiting for the full response. End-of-turn is signaled by response.audio.done.

5. Persist everything for post-call analytics

After the call ends, push the transcript and metadata to a queue so a GPT-4o-mini worker can extract sentiment, intent, and lead score without blocking the hot path.

async def on_call_end(call_id: str, transcript: list[dict]):
    await queue.publish("post_call", {"call_id": call_id, "transcript": transcript})

Production considerations

• Latency budget: target 800ms end-to-end. Allocate 150ms network, 200ms STT partial, 250ms LLM first token, 150ms TTS first frame, 50ms edge.
• Observability: emit an OpenTelemetry span for each stage with the call SID as the trace ID.
• Cost: Realtime minutes are the biggest line item. Hang up aggressively on silence and cap max session duration.
• Scale: one Python worker can handle 20-40 concurrent sessions before event-loop contention bites. Scale horizontally behind a sticky load balancer.
• Failure modes: if OpenAI returns 5xx mid-call, fall back to a canned "one moment please" and retry once before handing off to a human.

CallSphere's real implementation

CallSphere runs this exact architecture in production. The voice and chat agents use the OpenAI Realtime API with gpt-4o-realtime-preview-2025-06-03, server VAD, and PCM16 at 24kHz. Post-call analytics are handled by a GPT-4o-mini pipeline that writes sentiment, intent, and lead score into per-vertical Postgres databases. Telephony goes through Twilio with a WebRTC fallback for in-browser testing.

Each vertical has a different multi-agent topology: 14 tools for the healthcare voice stack, 10 agents for real estate (buyer, seller, rental, tour, qualification, and more), 4 for salon, 7 for after-hours escalation, 10 tools plus RAG for IT helpdesk, and a sales pod that pairs ElevenLabs TTS with 5 GPT-4 specialists. Handoffs between agents are orchestrated with the OpenAI Agents SDK. The platform supports 57+ languages, and end-to-end response times stay under 1 second on our production traffic.

Common pitfalls

• Buffering audio too long: you will hear obvious lag. Flush frames as soon as they arrive.
• Ignoring the VAD speech-started event: the agent will talk over interrupting callers.
• Sharing one HTTP client across calls improperly: connection pool exhaustion under load.
• Letting tool calls block the audio loop: always run tools in a separate task.
• Logging raw PCM: you will blow out disk. Log metadata only.
• Hardcoding a single voice: different verticals and languages need different voices; parameterize it.

FAQ

Why not stitch separate STT, LLM, and TTS services together?

You can, and some teams do, but each hop adds 100-300ms of latency and makes interruption handling much harder. The Realtime API collapses the pipeline into one WebSocket and gives you a clean speech-started signal for free.

What sample rate should I use?

24kHz PCM16 end to end. Convert to and from G.711 ulaw only at the carrier boundary. Resampling in the middle of the pipeline is a common source of audio artifacts.

How do I prevent the model from hallucinating facts about my business?

Constrain it with tool calls. The model should look up availability, prices, and policies through functions, not recall them from the system prompt.

What is a realistic concurrent-call number per worker?

With a tight async loop and no blocking tool calls, 20-40 sessions per Python worker is achievable. Beyond that, scale horizontally.

How do I handle a caller who speaks a different language than expected?

Detect the language from the first user turn and reload the session with the matching voice and instructions. CallSphere supports 57+ languages this way.

Next steps

Ready to see a real voice agent running this architecture? Book a demo, explore the technology page, or check pricing to understand how CallSphere packages this stack for production use.

#CallSphere #AIVoiceAgents #OpenAIRealtime #VoiceAI #Twilio #RealtimeAPI #TechnicalGuide