By Sagar Shankaran, Founder of CallSphere
Vendors quote 5% WER on benchmarks. In production AI voice, the same model hits 15% to 20% on noisy mobile calls and 50% in healthcare. Here is the live monitoring stack that catches drift before users notice.
Key takeaways
A vendor's 5% WER benchmark is a clean dictation in a quiet room. Your AI voice agent runs on cell-phone calls in salons with a hairdryer, in hospitals with intercoms, in cars at 70 mph. The same model that scored 5% on benchmarks scores 15-20% on real production traffic. Without continuous WER monitoring you will not know your accuracy until your conversion rate drops.
Vendors publish single-WER numbers from controlled audio. Real telephony adds 8 kHz narrowband, codec compression, background noise, accented speakers, and overlapping turns. Same provider can deliver 92% on headsets, 78% in conference rooms, 65% on noisy mobile (Deepgram's own data). Worse, WER drifts: a model update by your STT vendor can move WER by 3-5 points overnight without notice.
The second trap is using WER alone. WER counts word-level edits. "Schedule for tomorrow at 3" -> "Schedule for tomorrow at 2" is one substitution but a complete failure. Semantic Error Rate (SER) and intent-correctness are better signals for AI voice.
Sample 1-3% of calls per tenant. Run a high-quality reference model (Whisper-large-v3 or a paid commercial reference) on the recording and compare against your production STT output. Compute WER, character error rate, intent-match rate, and SER. Track P50/P95 per tenant per agent per day. Alert when 7-day rolling P50 WER moves >2 points or P95 moves >5 points.
flowchart TD
A[Production STT - per turn] --> B[Persist transcript + audio clip]
B --> C{Sample 1-3%?}
C -->|Yes| D[Run reference model - Whisper-large]
D --> E[Diff: WER, CER, SER, intent match]
E --> F[Persist accuracy_samples]
F --> G[Daily WER P50/P95 dashboard]
G --> H{Drift > 2pt?}
H -->|Yes| I[Alert - vendor regression]
CallSphere monitors STT accuracy live across all six verticals. Each of our 37 agents calls one of 90+ tools, and every transcript is persisted with confidence scores into 115+ DB tables. We sample 2% of calls per tenant per day, run Whisper-large-v3 as our reference, and store WER + SER + intent-match for trend analysis. Healthcare tenants on /industries/healthcare get 5% sampling for compliance. Twilio handles the carrier audio; Deepgram and OpenAI handle production STT; we monitor both. Starter ($149/mo) gets vendor WER alerts; Growth ($499/mo) and Scale ($1499/mo) get per-agent SER and intent-match. 14-day trial. Affiliates 22%.
Hear it before you finish reading
Talk to a live CallSphere AI voice agent in your browser — 60 seconds, no signup.
Why not just trust vendor confidence scores? Vendor confidence is well-calibrated for the vendor's own model but useless for cross-vendor comparison. Run a reference and compute true WER.
What sampling rate is enough? Deepgram recommends 30 minutes to 3 hours of audio for a stable WER baseline. 1-3% per tenant per day usually clears that threshold for active tenants.
What is good production WER? Below 10% is excellent, 10-15% is acceptable, 15-20% is degraded, above 20% needs intervention. Healthcare conversational WER often runs 25-50% even with strong models.
Should I switch vendors when WER spikes? Not always. Spikes often correlate to a code change in your turn-taking logic or audio pre-processing. Investigate before swapping.
How does SER differ from WER? WER counts edits at word level. SER counts whether meaning is preserved. "3 PM" -> "3 AM" is one word edit but a full failure. Embed both transcripts and compare cosine similarity.
Start a 14-day trial with WER monitoring, see pricing for SER on Growth, or book a demo. Healthcare on /industries/healthcare gets 5% sampling; partners earn 22% via the affiliate program.
ASR Accuracy Monitoring (WER) for Voice AI in Production - 2026 forces a tension most teams underestimate: agent handoff state. A single LLM call is easy. A booking agent that hands a confirmed slot to a billing agent that hands a follow-up to an escalation agent — that's where context loss, hallucinated IDs, and double-bookings live. Solving it well means treating the conversation as a stateful workflow, not a chat.
Still reading? Stop comparing — try CallSphere live.
CallSphere ships complete AI voice agents per industry — 14 tools for healthcare, 10 agents for real estate, 4 specialists for salons. See how it actually handles a call before you book a demo.
Production AI agents live or die on three loops: evals, retries, and handoff state. CallSphere runs 37 agents across 6 verticals, each with its own eval suite — synthetic call transcripts replayed nightly with assertion checks on extracted entities (date, time, party size, insurance, address). Without that loop, prompt regressions ship silently and you only find out when bookings drop.
Structured tools beat free-form text every time. Our 90+ function tools all enforce JSON schemas validated server-side; if the model hallucinates an integer where a string is required, we retry with a corrective system message before falling back to a deterministic path. For long-running flows, we treat agent handoffs as a state machine — booking → confirmation → SMS — so context survives turn boundaries.
The Realtime API vs. async decision usually comes down to "is the user holding the phone right now?" If yes, Realtime; if no (callback queue, after-hours voicemail), async wins on cost-per-conversation, which we track per agent in 115+ database tables spanning all 6 verticals.
What's the right way to scope the proof-of-concept?
Real Estate runs as a 6-container pod (frontend, gateway, ai-worker, voice-server, NATS event bus, Redis) backed by Postgres realestate_voice with row-level security so multi-tenant data never crosses tenants. For a topic like "ASR Accuracy Monitoring (WER) for Voice AI in Production - 2026", that means you're not starting from scratch — you're configuring an agent template that's already been hardened across thousands of conversations.
How do you handle compliance and data isolation? Day one is integration mapping (scheduler, CRM, messaging) and prompt tuning against your top 20 real call transcripts. Day two through five is shadow-mode running, where the agent transcribes and recommends but a human still answers, so you can compare side-by-side. Go-live is the moment your eval pass-rate clears your internal bar.
When does it make sense to switch from a managed model to a self-hosted one? The honest answer: it scales until your tool catalog gets stale. The agent is only as good as the integrations it can actually call, so the operational discipline is keeping schemas, webhooks, and fallback paths green. The platform handles the rest — observability, retries, multi-region routing — without your team owning the GPU layer.
Want to see how this maps to your stack? Book a live walkthrough at calendly.com/sagar-callsphere/new-meeting, or try the vertical-specific demo at salon.callsphere.tech. 14-day trial, no credit card, pilot live in 3–5 business days.
Written by
Sagar Shankaran· Founder, CallSphere
Sagar Shankaran is the founder of CallSphere, where he builds production AI voice and chat agents deployed across healthcare, hospitality, real estate, and home services. He writes about agentic AI, LLM engineering, and shipping voice agents that handle real calls in production.
See how AI voice agents work for your industry. Live demo available -- no signup required.
OpenAI's GPT-Realtime-Whisper launches at $0.017/min for streaming STT. Side-by-side latency, accuracy, and cost math vs Deepgram and the field.
How to actually observe a WebSocket fleet: ping/pong heartbeats, Prometheus metrics that matter, dead-man switches, and the alerts that fire before customers notice.
Run STT, LLM, and TTS entirely on Cloudflare's edge — no OpenAI, no ElevenLabs. Real working code with Whisper, Llama 3.3 70B, and Deepgram Aura.
The full metric set for evaluating production voice agents — STT word error rate, end-to-end latency budgets, RAG grounding, prosody, and the metrics that actually correlate with retention.
Build a working voice agent with the OpenAI Realtime API + Agents SDK, then bolt on an eval pipeline that catches barge-in failures, hallucinated grounding, and latency regressions.
Enterprise CIO Guide perspective on Aura 2 is Deepgram's TTS engine tuned for the latency and prosody constraints of real-time voice agents.
© 2026 CallSphere LLC. All rights reserved.