By Sagar Shankaran, Founder of CallSphere
How to build voice agents supporting 57+ languages using Deepgram, Whisper, ElevenLabs multilingual voices, real-time translation, and language detection patterns.
Key takeaways
Building a voice agent that speaks only English leaves 75% of the global market on the table. As of 2026, enterprises deploying voice AI across international operations need agents that handle at minimum 10-15 languages for European markets and 25-30 for global coverage. The leading platforms now support 50-60 languages, but raw language count is misleading — what matters is accuracy, latency, and naturalness per language.
This guide covers the architecture for building multilingual voice agents, the tradeoffs between different speech providers, language detection strategies, and real-time translation patterns for cross-language conversations.
The speech AI ecosystem offers varied levels of multilingual support. Here is the current landscape for production-ready language support:
flowchart LR
CALLER(["Caller"])
subgraph TEL["Telephony"]
SIP["Twilio SIP and PSTN"]
end
subgraph BRAIN["Business AI Agent"]
STT["Streaming STT<br/>Deepgram or Whisper"]
NLU{"Intent and<br/>Entity Extraction"}
TOOLS["Tool Calls"]
TTS["Streaming TTS<br/>ElevenLabs or Rime"]
end
subgraph DATA["Live Data Plane"]
CRM[("CRM and Notes")]
CAL[("Calendar and<br/>Schedule")]
KB[("Knowledge Base<br/>and Policies")]
end
subgraph OUT["Outcomes"]
O1(["Booking captured"])
O2(["CRM record created"])
O3(["Human handoff"])
end
CALLER --> SIP --> STT --> NLU
NLU -->|Lookup| TOOLS
TOOLS <--> CRM
TOOLS <--> CAL
TOOLS <--> KB
NLU --> TTS --> SIP --> CALLER
NLU -->|Resolved| O1
NLU -->|Schedule| O2
NLU -->|Escalate| O3
style CALLER fill:#f1f5f9,stroke:#64748b,color:#0f172a
style NLU fill:#4f46e5,stroke:#4338ca,color:#fff
style O1 fill:#059669,stroke:#047857,color:#fff
style O2 fill:#0ea5e9,stroke:#0369a1,color:#fff
style O3 fill:#f59e0b,stroke:#d97706,color:#1f2937
Speech-to-Text:
Text-to-Speech:
End-to-End:
The key decision is whether to use an end-to-end approach (simpler, fewer languages) or a composable pipeline (more complex, wider coverage).
Hear it before you finish reading
Talk to a live CallSphere AI voice agent in your browser — 60 seconds, no signup.
A multilingual voice agent needs to detect the caller's language, route to the appropriate STT model, reason in the detected language, and synthesize output in matching voice and language.
from dataclasses import dataclass
from enum import Enum
import asyncio
class LanguageTier(Enum):
TIER_1 = "tier_1" # Full support: native STT, LLM, TTS
TIER_2 = "tier_2" # Supported: may use translation bridge
TIER_3 = "tier_3" # Basic: translation-dependent
@dataclass
class LanguageConfig:
code: str # ISO 639-1 code
name: str
tier: LanguageTier
stt_provider: str
stt_model: str
tts_provider: str
tts_voice: str
llm_native: bool # Whether the LLM reasons natively in this language
# Language configuration registry
LANGUAGE_CONFIGS: dict[str, LanguageConfig] = {
"en": LanguageConfig(
code="en", name="English", tier=LanguageTier.TIER_1,
stt_provider="deepgram", stt_model="nova-2",
tts_provider="elevenlabs", tts_voice="rachel",
llm_native=True,
),
"es": LanguageConfig(
code="es", name="Spanish", tier=LanguageTier.TIER_1,
stt_provider="deepgram", stt_model="nova-2",
tts_provider="elevenlabs", tts_voice="maria",
llm_native=True,
),
"ja": LanguageConfig(
code="ja", name="Japanese", tier=LanguageTier.TIER_1,
stt_provider="deepgram", stt_model="nova-2",
tts_provider="elevenlabs", tts_voice="yuki",
llm_native=True,
),
"hi": LanguageConfig(
code="hi", name="Hindi", tier=LanguageTier.TIER_2,
stt_provider="whisper", stt_model="large-v3-turbo",
tts_provider="google", tts_voice="hi-IN-Wavenet-A",
llm_native=True,
),
"sw": LanguageConfig(
code="sw", name="Swahili", tier=LanguageTier.TIER_3,
stt_provider="whisper", stt_model="large-v3-turbo",
tts_provider="google", tts_voice="sw-TZ-Standard-A",
llm_native=False, # Use translation bridge
),
}
class MultilingualVoicePipeline:
def __init__(self):
self.stt_clients = {}
self.tts_clients = {}
self.translator = TranslationBridge()
async def process(
self, audio_stream, detected_language: str | None = None
):
# Step 1: Detect language if not known
if not detected_language:
detected_language = await self.detect_language(audio_stream)
config = LANGUAGE_CONFIGS.get(detected_language)
if not config:
config = LANGUAGE_CONFIGS["en"] # Fallback to English
# Step 2: Transcribe with language-specific STT
stt = self.get_stt_client(config)
transcript = await stt.transcribe(
audio_stream, language=config.code, model=config.stt_model
)
# Step 3: LLM reasoning (with translation bridge if needed)
if config.llm_native:
response = await self.llm_generate(transcript, language=config.code)
else:
# Translate to English, reason, translate back
en_transcript = await self.translator.translate(
transcript, source=config.code, target="en"
)
en_response = await self.llm_generate(en_transcript, language="en")
response = await self.translator.translate(
en_response, source="en", target=config.code
)
# Step 4: Synthesize with language-specific TTS
tts = self.get_tts_client(config)
audio = await tts.synthesize(
response, voice=config.tts_voice, language=config.code
)
return audio
The tier system is crucial. Tier-1 languages (English, Spanish, French, German, Japanese, Mandarin) get native STT, native LLM reasoning, and high-quality TTS with minimal latency. Tier-2 languages (Hindi, Arabic, Korean, Portuguese) may use slower STT models like Whisper but still get native LLM reasoning. Tier-3 languages (Swahili, Tagalog, Burmese) require a translation bridge where the LLM reasons in English and results are translated back.
Detecting the caller's language needs to happen in the first 1-3 seconds of audio. There are three approaches:
For phone-based agents, use the caller's phone number country code or IVR selection as a strong prior:
def predict_language_from_phone(phone_number: str) -> str:
"""Use phone number country code as language prior."""
country_code_map = {
"+1": "en", # US/Canada
"+44": "en", # UK
"+34": "es", # Spain
"+81": "ja", # Japan
"+91": "hi", # India (could also be en)
"+33": "fr", # France
"+49": "de", # Germany
}
for prefix, lang in sorted(
country_code_map.items(), key=lambda x: -len(x[0])
):
if phone_number.startswith(prefix):
return lang
return "en" # Default
This is fast (zero latency) but imprecise. A +1 number could be a Spanish speaker. Use it as a prior and confirm with audio-based detection.
Use a lightweight language identification model on the first 2-3 seconds of audio:
import whisper
import numpy as np
class AudioLanguageDetector:
def __init__(self):
self.model = whisper.load_model("base") # Small model for speed
async def detect(self, audio_chunk: np.ndarray) -> tuple[str, float]:
"""
Detect language from first 2-3 seconds of audio.
Returns (language_code, confidence).
"""
# Whisper's built-in language detection
audio = whisper.pad_or_trim(audio_chunk)
mel = whisper.log_mel_spectrogram(audio).to(self.model.device)
_, probs = self.model.detect_language(mel)
detected_lang = max(probs, key=probs.get)
confidence = probs[detected_lang]
return detected_lang, confidence
This adds 200-400ms of latency but is accurate. Run it in parallel with the initial STT processing — if the detected language differs from the assumed language, restart the STT connection with the correct language setting.
The production pattern combines both approaches and adds an explicit confirmation step for ambiguous cases:
async def determine_language(phone_number: str, initial_audio: bytes) -> str:
"""Multi-signal language detection with graceful fallback."""
# Signal 1: Phone number prior
phone_lang = predict_language_from_phone(phone_number)
# Signal 2: Audio-based detection
audio_lang, confidence = await audio_detector.detect(initial_audio)
# If both agree, high confidence
if phone_lang == audio_lang:
return audio_lang
# If audio detection is confident, trust it
if confidence > 0.85:
return audio_lang
# Ambiguous: use phone prior but prepare to switch
return phone_lang
Some use cases require the voice agent to converse in one language while executing business logic in another. For example, a Japanese caller interacting with a system where all product data is in English.
Still reading? Stop comparing — try CallSphere live.
CallSphere ships complete AI voice agents per industry — 14 tools for healthcare, 10 agents for real estate, 4 specialists for salons. See how it actually handles a call before you book a demo.
class TranslationBridge:
"""Real-time translation using LLM for high-quality contextual translation."""
def __init__(self, client):
self.client = client
self.context_buffer: list[dict] = []
async def translate(
self, text: str, source: str, target: str, domain: str = "general"
) -> str:
"""
Translate with conversation context for consistency.
Uses LLM for higher quality than dedicated translation APIs.
"""
# Include recent context for pronoun resolution and terminology consistency
context = "\n".join(
f"{m['lang']}: {m['text']}" for m in self.context_buffer[-4:]
)
response = await self.client.chat.completions.create(
model="gpt-4o-mini", # Fast and cheap for translation
messages=[
{
"role": "system",
"content": (
f"You are a real-time translator for a {domain} customer service conversation. "
f"Translate from {source} to {target}. "
"Preserve meaning, tone, and formality level. "
"Use domain-specific terminology where appropriate. "
"Output ONLY the translation, nothing else."
),
},
{
"role": "user",
"content": f"Context:\n{context}\n\nTranslate: {text}",
},
],
max_tokens=500,
temperature=0.3,
)
translated = response.choices[0].message.content.strip()
# Track context for consistency
self.context_buffer.append({"lang": source, "text": text})
self.context_buffer.append({"lang": target, "text": translated})
return translated
Using an LLM for translation instead of a dedicated translation API (Google Translate, DeepL) provides better contextual consistency. The LLM understands the conversation flow and maintains consistent terminology. The tradeoff is higher cost and 100-200ms additional latency per translation. For Tier-3 languages where this bridge is needed, the added latency is acceptable since these deployments already target 800-1200ms total response time.
Each language needs a voice that sounds native, not like an English speaker attempting the language. ElevenLabs handles this best with their multilingual voice cloning:
# Creating a consistent brand voice across languages with ElevenLabs
from elevenlabs import VoiceSettings
multilingual_voice_config = {
"en": {
"voice_id": "custom_brand_voice_en",
"settings": VoiceSettings(stability=0.75, similarity_boost=0.80),
},
"es": {
"voice_id": "custom_brand_voice_es", # Same base voice, Spanish clone
"settings": VoiceSettings(stability=0.70, similarity_boost=0.85),
},
"fr": {
"voice_id": "custom_brand_voice_fr",
"settings": VoiceSettings(stability=0.72, similarity_boost=0.82),
},
"ja": {
"voice_id": "yuki", # Use native Japanese voice for best results
"settings": VoiceSettings(stability=0.80, similarity_boost=0.75),
},
}
For languages where voice cloning is not available or quality is insufficient, use the provider's best native voice rather than a cloned version. A native-sounding Google WaveNet voice in Hindi is better than a poor ElevenLabs clone.
Testing multilingual agents requires native speakers — automated metrics miss cultural and linguistic nuances:
Maintain a test corpus of at least 200 utterances per supported language, covering accents, dialects, and speaking speeds representative of your user base.
Implement continuous language monitoring on the STT output. Run a lightweight language classifier on each transcribed sentence. When a language switch is detected with high confidence (>0.85), dynamically reconfigure the STT and TTS for the new language. The LLM typically handles code-switching naturally if the system prompt instructs it to respond in the user's current language.
Tier-1 languages (English, Spanish, French, German, Japanese, Mandarin) achieve 3-5% WER with Deepgram Nova-2 and near-native TTS quality. Tier-2 languages (Hindi, Arabic, Korean) achieve 6-10% WER and good TTS quality. Tier-3 languages (Swahili, Tagalog) can see 12-18% WER and less natural TTS. The translation bridge for Tier-3 languages adds another source of error — expect 85-90% meaning preservation compared to 97-99% for native Tier-1 processing.
For STT, use the best model per language. Deepgram Nova-2 excels for its supported 36 languages. For languages outside Deepgram's coverage, fall back to Whisper or Google Cloud Speech. For TTS, always use language-specific voices rather than one multilingual model — native voices sound dramatically better. For LLM reasoning, GPT-4o and Claude handle 50+ languages natively, so a single model works well for reasoning.
Tier-1 languages add zero cost over English since the same providers and models are used. Tier-2 languages may add 10-20% cost if a more expensive STT model (Whisper via API) is needed. Tier-3 languages with translation bridges add 30-50% cost due to the additional LLM translation calls. At scale, the cost is still dramatically lower than maintaining multilingual human agent teams.
#MultilingualAI #VoiceAgents #SpeechAPIs #LanguageSupport #Deepgram #Whisper #ElevenLabs #GlobalAI
Written by
Sagar Shankaran· Founder, CallSphere
LinkedInSagar Shankaran is the founder of CallSphere, where he builds production AI voice and chat agents deployed across healthcare, hospitality, real estate, and home services. He writes about agentic AI, LLM engineering, and shipping voice agents that handle real calls in production.
See how AI voice agents work for your industry. Live demo available -- no signup required.
Graphiti is the open-source temporal knowledge graph for AI agents in 2026. Learn how bi-temporal memory beats vector RAG for voice agents and long-running LLMs.
Self-correction is now a property of the model, not the framework. What that means for production agent reliability, voice/chat fallbacks, and CallSphere.
How to design a multi-agent system using MCP for tools and A2A for cross-vendor coordination, with a CallSphere voice agent as a participating node.
OpenAI's GPT-Realtime-Whisper launches at $0.017/min for streaming STT. Side-by-side latency, accuracy, and cost math vs Deepgram and the field.
Build a working voice agent with the OpenAI Realtime API + Agents SDK, then bolt on an eval pipeline that catches barge-in failures, hallucinated grounding, and latency regressions.
Offline evals catch regressions before deploy on a fixed dataset. Online evals catch real-world drift on live traffic. You need both — here is how we run them.
© 2026 CallSphere Inc. All rights reserved.
Made within San Francisco
Watch how CallSphere handles real customer calls, schedules appointments, and processes payments — live.
Try Live DemoBook a DemoCalculate Your ROI