Cold Start vs Warm Start: First-Turn Latency for AI Voice (2026)

TL;DR — Cold starts on serverless GPU stretch first-turn latency from 600ms to 5-20 seconds. The fix is warm pools — keep enough idle instances loaded with the model to absorb concurrency spikes. For voice AI, the cost of an idle GPU is tiny vs the cost of a hung-up caller.

The latency problem

The first call after a deploy or a low-traffic window pays the cold-start tax: container pull, model weight load (5-30GB), KV cache initialization, WebSocket handshakes. A chatbot that usually replies in <1s can take 5-20s on cold start; voice agents simply hang up first.

Where the ms come from

Cold-start cost decomposes into:

1. Container start — 1-5s (image pull + cgroup setup)
2. Model weight load — 2-15s (depends on size + storage class)
3. CUDA / GPU init — 0.5-2s
4. Warmup forward pass — 100-500ms (so first inference doesn't recompile kernels)
5. WebSocket / SIP registration — 100-500ms

Total first-turn cost: 5-25s on a true cold start. For voice, that's catastrophic.

flowchart LR
 CALL[Incoming call] --> POOL{Warm pool<br/>has capacity?}
 POOL -->|Yes| ROUTE[Route to warm<br/>~50ms]
 POOL -->|No| COLD[Cold start<br/>5-20s]
 ROUTE --> TURN[First turn<br/>= 600ms]
 COLD --> WARMUP[Load model<br/>+ warmup]
 WARMUP --> TURN

CallSphere stack

Explore a live demo and compare current plans to find the right fit for your business.

Explore a live demo and compare current plans to find the right fit for your business.

Optimization steps

1. Set a non-zero minimum replica count on every voice-serving deployment. Zero-scale is for batch jobs, not voice.
2. Pre-warm GPU containers with a synthetic forward pass at boot, before they accept traffic.
3. Use provisioned concurrency on serverless GPU (Modal, Cerebrium, Replicate) — pay for idle capacity, save customers.
4. Pre-establish WebSocket pools to TTS/ASR vendors at instance start, not at first turn.
5. Monitor "first-turn vs steady-state" latency separately — they have different failure modes.

FAQ

Q: How much does a warm pool cost? 1-2 always-on GPUs per region. Cheaper than the churn from a hung-up caller.

Q: Does Realtime API cold-start? Effectively no for the model itself, but your client-side WebSocket setup still pays a 200-500ms handshake.

Q: Should I use serverless for voice? Only with provisioned concurrency. Pure on-demand serverless is incompatible with sub-second voice.

Q: How does CallSphere handle traffic bursts? Auto-scales above the warm baseline; new instances pre-warm in shadow mode before joining the load balancer.

Q: What about regional failover? Warm replicas in 2+ regions; DNS failover with 30s TTL on health-check failure.

Sources

• StrongMocha — Why Warm Pools Beat Cold Starts
• DigitalOcean — Hidden Cost of Cold Starts in Serverless AI
• Digital Applied — Voice Agent Infrastructure Stack 2026
• Plivo — Best AI Voice Agents for Cold Calling 2026

Cold Start vs Warm Start: First-Turn Latency for AI Voice (2026): production view

Cold Start vs Warm Start: First-Turn Latency for AI Voice (2026) forces a tension most teams underestimate: agent handoff state. A single LLM call is easy. A booking agent that hands a confirmed slot to a billing agent that hands a follow-up to an escalation agent — that's where context loss, hallucinated IDs, and double-bookings live. Solving it well means treating the conversation as a stateful workflow, not a chat.

Serving stack tradeoffs

The big fork is managed (OpenAI Realtime, ElevenLabs Conversational AI) versus self-hosted on GPUs you operate. Managed wins on cold-start, model freshness, and zero-ops; self-hosted wins on unit economics past a certain conversation volume and on data residency for regulated verticals. CallSphere runs hybrid: Realtime for live calls, self-hosted Whisper + a hosted LLM for async, both routed through a Go gateway that enforces per-tenant rate limits.

Latency budgets are non-negotiable on voice. End-to-end target is sub-800ms ASR-to-first-token and sub-1.4s first-audio-out; anything beyond that and turn-taking feels stilted. GPU residency in the same region as your TURN servers matters more than choosing a slightly bigger model.

Observability is the unglamorous backbone — every conversation produces logs, traces, sentiment scoring, and cost attribution piped to a per-tenant dashboard. HIPAA aligned isolation keeps healthcare traffic separated from salon traffic at the storage layer, not just the API.

FAQ

What's the right way to scope the proof-of-concept? Real Estate runs as a 6-container pod (frontend, gateway, ai-worker, voice-server, NATS event bus, Redis) backed by Postgres realestate_voice with row-level security so multi-tenant data never crosses tenants. For a topic like "Cold Start vs Warm Start: First-Turn Latency for AI Voice (2026)", that means you're not starting from scratch — you're configuring an agent template that's already been hardened across thousands of conversations.

How do you handle compliance and data isolation? Day one is integration mapping (scheduler, CRM, messaging) and prompt tuning against your top 20 real call transcripts. Day two through five is shadow-mode running, where the agent transcribes and recommends but a human still answers, so you can compare side-by-side. Go-live is the moment your eval pass-rate clears your internal bar.

When does it make sense to switch from a managed model to a self-hosted one? The honest answer: it scales until your tool catalog gets stale. The agent is only as good as the integrations it can actually call, so the operational discipline is keeping schemas, webhooks, and fallback paths green. The platform handles the rest — observability, retries, multi-region routing — without your team owning the GPU layer.

Talk to us

Explore a live demo and compare current plans to find the right fit for your business.