ONNX Runtime + WebGPU for Browser Voice Agents (No Server, Sub-100ms)

TL;DR — ONNX Runtime Web + WebGPU has matured enough in 2026 to run Whisper-tiny + Kokoro + LFM2.5-Audio-1.5B entirely in the browser tab. Recent updates: Flash Attention, graph capture, Split-K MatMul, qMoE support. WebGPU works in Chrome, Edge, Safari (16+), Firefox Nightly. Result: a voice agent that sends zero audio to your servers — perfect for HIPAA, GDPR, or pure latency.

Why browser inference for voice

• Zero network round-trip for STT/TTS — only the LLM tokens (or none, if you run a small SLM locally).
• Privacy by construction — audio never leaves the device.
• Cost — your users pay for the GPU.
• Offline — works on a plane.

Architecture

flowchart LR
 MIC[Mic Stream] --> ORT[ONNX Runtime Web]
 ORT -->|WebGPU| STT[Whisper-tiny INT8]
 STT -->|text| LLM{LLM Choice}
 LLM -->|local| LFM[LFM2.5-Audio 1.5B in-browser]
 LLM -->|remote| API[Cloudflare / Groq]
 LFM & API -->|tokens| TTS[Kokoro-82M WebGPU]
 TTS -->|PCM| OUT[Audio Output]

CallSphere stack with ONNX Web

Explore a live demo and compare current plans to find the right fit for your business.

Build steps

1. npm install onnxruntime-web@1.21 (latest 2026 build with WebGPU EP improvements).
2. Convert Whisper-tiny to ONNX with optimum-cli export onnx --model openai/whisper-tiny --task automatic-speech-recognition.
3. Quantize to INT8 with onnxruntime.quantization for 4× smaller download.
4. In your page: const session = await ort.InferenceSession.create(url, { executionProviders: ['webgpu'] });.
5. Use Web Audio API AudioWorklet to grab 16kHz mono frames; feed into Whisper session.
6. Pipe Whisper output to remote LLM (Workers AI is closest), or to in-browser LFM2.5-Audio for true offline.
7. Pipe LLM tokens into Kokoro ONNX session; play via AudioBufferSourceNode.

Pitfalls

• WebGPU adoption — Safari 16+, Chrome 113+, but mobile Safari has shader-compile bugs as of 2026. Test on real devices.
• Model size — Whisper-tiny is 39M params (≈80MB INT8). Kokoro 82M is ≈170MB. Lazy-load.
• First-token latency — Initial WebGPU pipeline compile is 2–5s; warm with a silent decode on page load.
• Memory pressure on iOS — keep total VRAM use under 1GB or iOS will kill the tab.
• Microphone permissions — UX matters; explain why before asking.

FAQ

Q: Can I run Llama in-browser? A: Yes — LFM2.5-Audio-1.5B (Liquid AI) and Phi-3.5-mini run on WebGPU. Speeds: 15–30 tok/s on M2 MacBook.

Q: Whisper-large? A: Too big (1.6GB). Stick with tiny/base in-browser; route long-form to a server.

Q: HIPAA? A: In-browser inference means audio never leaves the device — ideal for /industries/healthcare.

Q: Mobile? A: WebGPU on iOS 17+; Android Chrome 113+. Performance varies wildly by GPU.

Q: Cost? A: Free at runtime — users pay. CallSphere widget is included on Orchestrate plan and above (/pricing).

Sources

• ONNX Runtime WebGPU docs
• ONNX Runtime Web NPM
• WebGPU + ORT inference guides (Modexa)
• State of On-Device AI in Browser
• Liquid LFM2.5 cookbook

ONNX Runtime + WebGPU for Browser Voice Agents (No Server, Sub-100ms): production view

ONNX Runtime + WebGPU for Browser Voice Agents (No Server, Sub-100ms) ultimately resolves into one engineering question: when do you use the OpenAI Realtime API versus an async pipeline? Realtime wins on latency for live calls. Async wins on cost, retries, and structured tool reliability for callbacks and SMS flows. Most teams need both, and the routing layer between them becomes the most load-bearing piece of the stack.

Shipping the agent to production

Production AI agents live or die on three loops: evals, retries, and handoff state. CallSphere runs 37 agents across 6 verticals, each with its own eval suite — synthetic call transcripts replayed nightly with assertion checks on extracted entities (date, time, party size, insurance, address). Without that loop, prompt regressions ship silently and you only find out when bookings drop.

Structured tools beat free-form text every time. Our 90+ function tools all enforce JSON schemas validated server-side; if the model hallucinates an integer where a string is required, we retry with a corrective system message before falling back to a deterministic path. For long-running flows, we treat agent handoffs as a state machine — booking → confirmation → SMS — so context survives turn boundaries.

The Realtime API vs. async decision usually comes down to "is the user holding the phone right now?" If yes, Realtime; if no (callback queue, after-hours voicemail), async wins on cost-per-conversation, which we track per agent in 115+ database tables spanning all 6 verticals.

FAQ

Why does onnx runtime + webgpu for browser voice agents (no server, sub-100ms) matter for revenue, not just engineering? 57+ languages are supported out of the box, and the platform is HIPAA aligned, which removes most of the procurement friction in regulated verticals. For a topic like "ONNX Runtime + WebGPU for Browser Voice Agents (No Server, Sub-100ms)", that means you're not starting from scratch — you're configuring an agent template that's already been hardened across thousands of conversations.

What are the most common mistakes teams make on day one? Day one is integration mapping (scheduler, CRM, messaging) and prompt tuning against your top 20 real call transcripts. Day two through five is shadow-mode running, where the agent transcribes and recommends but a human still answers, so you can compare side-by-side. Go-live is the moment your eval pass-rate clears your internal bar.

How does CallSphere's stack handle this differently than a generic chatbot? The honest answer: it scales until your tool catalog gets stale. The agent is only as good as the integrations it can actually call, so the operational discipline is keeping schemas, webhooks, and fallback paths green. The platform handles the rest — observability, retries, multi-region routing — without your team owning the GPU layer.

Talk to us

Explore a live demo and compare current plans to find the right fit for your business.