Capacity Planning for Voice AI: Concurrent Calls, GPU Headroom, and Burst Math

TL;DR — Forecast peak concurrent calls, multiply by 1.5x burst factor, then size workers, model TPM, and GPU memory against that number. Most teams under-size by 3x.

What goes wrong

flowchart TD
 Client[Client] --> Edge[Cloudflare Worker]
 Edge -->|WS upgrade| DO[Durable Object]
 DO --> AI[(OpenAI Realtime WS)]
 AI --> DO
 DO --> Client
 DO -.hibernation.-> Storage[(Persisted state)]

Voice AI concurrent-call capacity is the metric that decides whether your customers hit busy signals on a Monday morning. Different platforms in 2026 land at wildly different ceilings: LuMay at 10k concurrent, Cognigy at tens of thousands across 100+ languages, Bland AI at 20k calls/hour, Synthflow's starter tier at 5 concurrent. A single agency client running a campaign generates 20–50 concurrent in a peak hour; ten clients means hundreds.

Three things break under capacity pressure:

1. WebSocket layer — limits on FDs per process, ulimits, kernel TCP backlogs.
2. Model TPM — OpenAI's tokens-per-minute caps; Realtime sessions count against them differently.
3. GPU memory — for self-hosted ASR/TTS, each session pins KV cache and audio buffers.

How to monitor

Track three capacity metrics:

1. Active concurrent sessions — gauge per vertical.
2. Headroom — concurrent / capacity_ceiling. Alert at 70% sustained for 10 min.
3. Burst factor — peak / median over 28 days. Use as multiplier for capacity planning.

Plan for a 1.5x burst over forecast peak, with a 2x growth buffer for the next quarter.

CallSphere stack

CallSphere is sized for ~3000 concurrent sessions across 6 verticals on a k3s cluster behind Cloudflare Tunnel. Per-vertical ceilings:

• Healthcare FastAPI :8084 — 800 concurrent. Each session uses ~120MB RAM, ~0.05 vCPU. 4 replicas × 4 vCPU × 16GB.
• Real Estate 6-container NATS pod — 600 concurrent (NATS message-per-call cap is the binding constraint). 4 replicas of the pod.
• Sales WebSocket + PM2 — 1000 concurrent. 8 PM2 workers per node × 4 nodes. ulimit set to 65k FDs.
• After-hours Bull/Redis queue — async, capacity is queue depth (cap 50k jobs). Scales differently because not real-time.

OpenAI Realtime caps: we're at 20M tokens/min on the production org; one minute of voice = ~6000 tokens, so theoretical ceiling is 3300 concurrent at full duplex. Actual real ceiling is lower because of headroom.

37 agents and 90+ tools means each session may invoke 5+ tool calls; we run tool executors as a horizontally scalable service (Sales WebSocket layer doubles as the tool runner).

Explore a live demo and compare current plans to find the right fit for your business.

Implementation

1. Forecast peak from history. Pull 13 weeks of call_started events; compute weekly p99 concurrent. That's your forecast peak.

2. Right-size per-pod.

# k3s deployment
spec:
 replicas: 4
 template:
 spec:
 containers:
 - name: healthcare
 resources:
 requests: { cpu: "2", memory: "8Gi" }
 limits: { cpu: "4", memory: "16Gi" }

3. HPA on concurrent_sessions metric.

metrics:
 - type: Pods
 pods:
 metric: { name: active_sessions_per_pod }
 target: { type: AverageValue, averageValue: "200" }

4. Pre-provision burst. During known peaks (Monday 9–11am for Sales), pre-scale via a CronJob to 1.5x baseline.

5. Reserve model capacity. OpenAI Provisioned Throughput (PTU) for the $149 Custom Build tier guarantees TPM. $49/$99 share burst capacity.

FAQ

Q: How do I forecast new growth? A: Forecast at the customer-onboarding level, not aggregate. Each new customer ships an estimated load profile.

Q: What's the right HPA metric? A: Active session count, not CPU. CPU lags reality by 30 seconds in voice.

Q: Can I oversubscribe model TPM? A: A little (10%). Beyond that you risk 429s mid-call.

Q: How do I cap burst per customer? A: Per-tenant rate limit at the WebSocket gateway. Returns a polite message and falls back to a queue.

Q: GPU vs API for self-hosted? A: API for < 1k concurrent. GPU when reserved capacity becomes cheaper than per-token. Math at our scale: cross-over around 2.5k sustained concurrent.

Sources

• Trillet — Voice AI Concurrent Call Capacity 2026
• Retell AI — AI Voice Agents in 2026
• Lindy — 18 Top AI Voice Agents 2026
• Voice.ai — Best Contact Center Infrastructure Software

Capacity Planning for Voice AI: Concurrent Calls, GPU Headroom, and Burst Math: production view

Capacity Planning for Voice AI: Concurrent Calls, GPU Headroom, and Burst Math sounds like a single decision, but in production it splits into eval design, prompt cost, and observability. The deeper you push toward live traffic, the more those three pull against each other — better evals catch silent failures, prompt cost limits how often you can re-run them, and weak observability hides which retries are actually saving conversations versus burning latency budget.

Serving stack tradeoffs

The big fork is managed (OpenAI Realtime, ElevenLabs Conversational AI) versus self-hosted on GPUs you operate. Managed wins on cold-start, model freshness, and zero-ops; self-hosted wins on unit economics past a certain conversation volume and on data residency for regulated verticals. CallSphere runs hybrid: Realtime for live calls, self-hosted Whisper + a hosted LLM for async, both routed through a Go gateway that enforces per-tenant rate limits.

Latency budgets are non-negotiable on voice. End-to-end target is sub-800ms ASR-to-first-token and sub-1.4s first-audio-out; anything beyond that and turn-taking feels stilted. GPU residency in the same region as your TURN servers matters more than choosing a slightly bigger model.

Observability is the unglamorous backbone — every conversation produces logs, traces, sentiment scoring, and cost attribution piped to a per-tenant dashboard. HIPAA aligned isolation keeps healthcare traffic separated from salon traffic at the storage layer, not just the API.

FAQ

How does this apply to a CallSphere pilot specifically? CallSphere runs 37 production agents and 90+ function tools across 115+ database tables in 6 verticals, so most workflows you'd want already have a template. For a topic like "Capacity Planning for Voice AI: Concurrent Calls, GPU Headroom, and Burst Math", that means you're not starting from scratch — you're configuring an agent template that's already been hardened across thousands of conversations.

What does the typical first-week implementation look like? Day one is integration mapping (scheduler, CRM, messaging) and prompt tuning against your top 20 real call transcripts. Day two through five is shadow-mode running, where the agent transcribes and recommends but a human still answers, so you can compare side-by-side. Go-live is the moment your eval pass-rate clears your internal bar.

Where does this break down at scale? The honest answer: it scales until your tool catalog gets stale. The agent is only as good as the integrations it can actually call, so the operational discipline is keeping schemas, webhooks, and fallback paths green. The platform handles the rest — observability, retries, multi-region routing — without your team owning the GPU layer.

Talk to us

Explore a live demo and compare current plans to find the right fit for your business.