By Sagar Shankaran, Founder of CallSphere
Build a voice agent that runs Whisper, Deepgram Flux, Aura-1 TTS, and Llama 3.3 on Cloudflare's 330-city edge network. Architecture, code, latency budget, and CallSphere-grade pitfalls.
Key takeaways
TL;DR — Cloudflare Workers AI hosts Deepgram Nova-3, Flux, Aura-1, Whisper-large-v3-turbo, MiniMax Speech 2.8 Turbo, and Llama 3.3 70B on a 330-city edge fabric with WebSocket support. Pair it with Cloudflare Realtime (WebRTC + SFU) and you get a voice agent whose every hop — STT, LLM, TTS, media — terminates in the same data center the caller dials into. Real-world TTFB under 200ms; full voice-to-voice round-trip 350–500ms.
Voice agents have a hard floor: humans perceive any silence over ~500ms as awkward. Stack a cloud STT round-trip (180ms), a region-locked LLM (220ms), and a TTS first-byte (300ms) and you blow the budget before the model even speaks. Cloudflare's 2026 play is to terminate all of that — WebRTC media, STT, LLM, TTS, and tool-calling — at the same edge node the SIP gateway or browser connects to. No public-internet hops, no cross-region latency tax.
flowchart LR
CALLER[PSTN / Browser] -->|WebRTC / SIP| EDGE[Cloudflare Edge POP]
EDGE --> RT[Realtime SFU]
RT -->|PCM frames| W1[Worker: STT - Deepgram Flux]
W1 -->|transcript| W2[Worker: LLM - Llama 3.3 70B]
W2 -->|tool calls| AG[AI Gateway + MCP]
W2 -->|reply text| W3[Worker: TTS - Aura-1]
W3 -->|audio frames| RT
RT -->|RTP| CALLER
CallSphere ships 37 production agents · 90+ tools · 115+ database tables · 6 verticals. On the Cloudflare path we map: 8 STT/turn-detection workers, 7 TTS workers, 14 LLM-orchestrator workers, and 8 tool-router workers — all bound to the same Durable Object per call so state never leaves the edge. Pricing tiers: Starter $149/mo, Growth $499/mo, Scale $1,499/mo with a 14-day trial and a 22% recurring affiliate split.
wrangler init voice-edge, add compatibility_date = "2026-03-01" and [ai] binding.@cf/deepgram/flux (turn-aware ASR purpose-built for voice agents). Buffer 20ms frames in.is_final segment, invoke @cf/meta/llama-3.3-70b-instruct via the AI binding; keep history in a Durable Object so reconnects survive.@cf/deepgram/aura-1 (sub-200ms TTFB) and write PCM back to the SFU.wrangler deploy. The same code runs in Mumbai, São Paulo, and Frankfurt simultaneously.exceeded CPU limit mid-conversation.placement.mode = "smart" only after measuring.Q: Does Workers AI support OpenAI Realtime API? A: Not natively. Use Cloudflare Realtime + Workers AI Llama for the LLM step, or proxy OpenAI Realtime through a Worker (you lose the edge-locality benefit).
Q: HIPAA?
A: Cloudflare signs BAAs on Enterprise. Workers AI inference logs can be disabled per-request with gateway: { collectLog: false }. See /industries/healthcare.
Hear it before you finish reading
Talk to a live CallSphere AI voice agent in your browser — 60 seconds, no signup.
Q: Cost vs OpenAI? A: Workers AI Llama 3.3 70B is ~$0.59/M input tokens vs $2.50 for OpenAI gpt-4o-mini at comparable quality — and zero egress.
Q: How does CallSphere use this? A: Our /demo routes US-East callers through Workers AI Aura-1 + Llama 3.3 for sub-400ms voice-to-voice. Start a /trial to test.
Q: Affiliate? A: 22% recurring on every customer you refer to /pricing via /affiliate.
Cloudflare Workers AI for Sub-100ms Voice Agents (2026 Edge Inference) sounds like a single decision, but in production it splits into eval design, prompt cost, and observability. The deeper you push toward live traffic, the more those three pull against each other — better evals catch silent failures, prompt cost limits how often you can re-run them, and weak observability hides which retries are actually saving conversations versus burning latency budget.
The big fork is managed (OpenAI Realtime, ElevenLabs Conversational AI) versus self-hosted on GPUs you operate. Managed wins on cold-start, model freshness, and zero-ops; self-hosted wins on unit economics past a certain conversation volume and on data residency for regulated verticals. CallSphere runs hybrid: Realtime for live calls, self-hosted Whisper + a hosted LLM for async, both routed through a Go gateway that enforces per-tenant rate limits.
Still reading? Stop comparing — try CallSphere live.
CallSphere ships complete AI voice agents per industry — 14 tools for healthcare, 10 agents for real estate, 4 specialists for salons. See how it actually handles a call before you book a demo.
Latency budgets are non-negotiable on voice. End-to-end target is sub-800ms ASR-to-first-token and sub-1.4s first-audio-out; anything beyond that and turn-taking feels stilted. GPU residency in the same region as your TURN servers matters more than choosing a slightly bigger model.
Observability is the unglamorous backbone — every conversation produces logs, traces, sentiment scoring, and cost attribution piped to a per-tenant dashboard. HIPAA + SOC 2 aligned isolation keeps healthcare traffic separated from salon traffic at the storage layer, not just the API.
How does this apply to a CallSphere pilot specifically? CallSphere runs 37 production agents and 90+ function tools across 115+ database tables in 6 verticals, so most workflows you'd want already have a template. For a topic like "Cloudflare Workers AI for Sub-100ms Voice Agents (2026 Edge Inference)", that means you're not starting from scratch — you're configuring an agent template that's already been hardened across thousands of conversations.
What does the typical first-week implementation look like? Day one is integration mapping (scheduler, CRM, messaging) and prompt tuning against your top 20 real call transcripts. Day two through five is shadow-mode running, where the agent transcribes and recommends but a human still answers, so you can compare side-by-side. Go-live is the moment your eval pass-rate clears your internal bar.
Where does this break down at scale? The honest answer: it scales until your tool catalog gets stale. The agent is only as good as the integrations it can actually call, so the operational discipline is keeping schemas, webhooks, and fallback paths green. The platform handles the rest — observability, retries, multi-region routing — without your team owning the GPU layer.
Want to see how this maps to your stack? Book a live walkthrough at calendly.com/sagar-callsphere/new-meeting, or try the vertical-specific demo at healthcare.callsphere.tech. 14-day trial, no credit card, pilot live in 3–5 business days.
Written by
Sagar Shankaran· Founder, CallSphere
Sagar Shankaran is the founder of CallSphere, where he builds production AI voice and chat agents deployed across healthcare, hospitality, real estate, and home services. He writes about agentic AI, LLM engineering, and shipping voice agents that handle real calls in production.
See how AI voice agents work for your industry. Live demo available -- no signup required.
OpenAI's GPT-Realtime-Whisper launches at $0.017/min for streaming STT. Side-by-side latency, accuracy, and cost math vs Deepgram and the field.
On May 4 2026 OpenAI published its Realtime stack rebuild — split-relay plus transceiver edge. Here is what changed and what it means for production voice agents.
Each Cloudflare agent runs on a Durable Object with its own SQLite, WebSockets, and scheduling. Agents Week 2026 shipped MCP, Code Mode, and 10GB SQLite per agent.
Enterprise CIO Guide perspective on Aura 2 is Deepgram's TTS engine tuned for the latency and prosody constraints of real-time voice agents.
The three real-time ASR engines competing for production voice-agent traffic in 2026, benchmarked on accuracy, latency, and cost.
SMB Founder Playbook perspective on Aura 2 is Deepgram's TTS engine tuned for the latency and prosody constraints of real-time voice agents.
© 2026 CallSphere LLC. All rights reserved.