Voice Agent Barge-In Handling: Server VAD, Client VAD, and the Hybrid Approach

What Barge-In Actually Is

Barge-in is the user interrupting the agent mid-sentence. Done right, the agent stops talking, listens, and responds to the new utterance. Done wrong — and most 2024 voice agents got it wrong — the agent talks over the user, ignores the interruption, or hallucinates a response combining their own half-finished output with the user's input.

Three patterns ship in production in 2026: server VAD, client VAD, and hybrid. Each has tradeoffs. Most teams pick one and learn the hard way that they should have picked the other.

The Three Patterns

flowchart TB
    subgraph Server[Server VAD]
        S1[Caller audio] --> S2[Stream to server]
        S2 --> S3[Server detects speech]
        S3 --> S4[Server cancels TTS]
    end
    subgraph Client[Client VAD]
        C1[Caller audio] --> C2[Local VAD]
        C2 --> C3[Send interrupt signal]
        C3 --> C4[Server cancels TTS]
    end
    subgraph Hybrid[Hybrid]
        H1[Local fast VAD] --> H2[Local interrupt + send signal]
        H2 --> H3[Server semantic VAD<br/>confirms]
        H3 --> H4[Commit cancel<br/>or resume]
    end

Server VAD

Audio streams to the server. The server detects speech, decides if it is an interruption, and cancels in-flight TTS. This is what OpenAI's Realtime API and most cloud voice services default to.

• Pro: simpler client; server has more compute and better models
• Con: round-trip means 100-300ms of TTS keeps playing after the user starts talking

Client VAD

A small VAD model (Silero, WebRTC VAD, or a lightweight transformer) runs on the device. When it detects speech, it sends an interrupt signal to the server.

• Pro: lowest latency to interruption — typically 50-100ms
• Con: false-positive interrupts from coughs, ambient noise, side conversations

Hybrid

Local VAD fires immediately and pauses TTS playback locally. The server's semantic VAD evaluates the audio and either confirms the interrupt (TTS stays cancelled) or resumes (TTS continues from where it paused).

• Pro: best of both — fast response, low false-positive rate
• Con: more complex; requires resumable TTS, which not all S2S models support

What Goes Wrong in Each

The failure modes that hit production:

• Server VAD over PSTN: cellular jitter and packet loss make end-of-utterance detection unreliable; the system either cuts the user off or waits too long
• Client VAD in noisy environments: a TV in the background triggers an interrupt mid-sentence
• Hybrid with non-resumable TTS: the cancel-and-resume race condition where the server thinks "resume" but the TTS already finished produces stuttering output

Tuning the Endpoint Threshold

flowchart LR
    Speech[Caller speaks] --> Detect{Above threshold?}
    Detect -->|Yes| Hold[Hold for X ms]
    Hold --> Confirm{Still speaking?}
    Confirm -->|Yes| Interrupt[Trigger interrupt]
    Confirm -->|No| Ignore[Ignore]

The hold-duration X is the most-tuned parameter in production voice agents. Too short (50ms) gives false positives from breaths. Too long (300ms) gives sluggish interrupts. The sweet spot for typical telephony agents is 80-150ms with semantic VAD, or 150-250ms with energy-based VAD.

What Native S2S Models Do Differently

GPT-4o-realtime, Gemini Live, and Sesame Maya all expose server VAD as the default and provide explicit response.cancel events for client-driven interrupts. The 2026 best practice with these models is to use server VAD as the floor and add client-side input-buffer cancellation for the cases where the round-trip cost is too high (long-form TTS, multi-sentence responses).

A Concrete Hybrid Implementation

sequenceDiagram
    participant Caller
    participant Client
    participant Server
    participant TTS
    Caller->>Client: starts speaking
    Client->>Client: local VAD fires (60ms)
    Client->>Server: interrupt signal
    Client->>Client: pause local TTS playback
    Server->>Server: semantic VAD evaluates 200ms
    alt confirmed interrupt
        Server->>TTS: cancel
        Server->>Client: confirm cancel
    else false positive
        Server->>Client: resume
        Client->>Client: resume playback
    end

This is the design we use on CallSphere's voice agents. False-positive rate dropped from 11 percent (server-only VAD) to 2.4 percent (hybrid) with no measurable increase in interrupt latency.

Sources

• Silero VAD — https://github.com/snakers4/silero-vad
• WebRTC VAD specification — https://webrtc.org
• OpenAI Realtime API VAD docs — https://platform.openai.com/docs/guides/realtime
• LiveKit barge-in patterns — https://docs.livekit.io
• "Endpoint detection in conversational AI" Deepgram — https://deepgram.com/learn

How this plays out in production

One layer below what Voice Agent Barge-In Handling: Server VAD, Client VAD, and the Hybrid Approach covers, the practical question every team hits is multi-turn handoffs between specialist agents without losing slot state, sentiment, or escalation context. Treat this as a voice-first system from the first prompt: the agent's persona, its tool surface, and its escalation rules all flow from that single decision. Teams that ship fast tend to instrument the loop end-to-end before they tune any single component, because the bottleneck is rarely where intuition puts it.

Voice agent architecture, end to end

A production-grade voice stack at CallSphere stitches Twilio Programmable Voice (PSTN ingress, TwiML, bidirectional Media Streams) to a realtime reasoning layer — typically OpenAI Realtime or ElevenLabs Conversational AI — with sub-second response as a hard SLO. Anything north of one second of perceived silence and callers either repeat themselves or hang up; that single number drives the whole architecture. Server-side VAD with proper barge-in support is non-negotiable, otherwise the agent talks over the caller and the conversation collapses. Streaming TTS with phoneme-aligned interruption keeps the cadence natural even when the user changes their mind mid-sentence. Post-call, every transcript is run through a structured pipeline: sentiment, intent classification, lead score, escalation flag, and a normalized slot extraction (name, callback number, reason, urgency). For healthcare workloads, the BAA-covered storage path, audit logs, encryption-at-rest, and PHI-safe transcript redaction are wired in from day one, not bolted on at compliance review. The end state is a system where every call produces a row of structured data, not just a recording.

FAQ

How do you actually ship a voice agent the way Voice Agent Barge-In Handling: Server VAD, Client VAD, and the Hybrid Approach describes?

Treat the architecture in this post as a starting point and instrument it before you tune it. The metrics that matter most early on are end-to-end latency (target < 1s for voice, < 3s for chat), barge-in correctness, tool-call success rate, and post-conversation lead score distribution. Optimize whatever the data flags as the bottleneck, not whatever feels slowest in your head.

What are the failure modes of voice agent deployments at scale?

The two failure modes that bite hardest are silent context loss across multi-turn handoffs and tool calls that succeed in dev but get rate-limited in production. Both are solvable with a proper agent backplane that pins state to a session ID, retries with backoff, and writes every tool invocation to an audit log you can replay.

What does the CallSphere outbound sales calling product do that a regular dialer does not?

It uses the ElevenLabs "Sarah" voice, runs up to 5 concurrent outbound calls per operator, and ships with a browser-based dialer that transfers warm calls back to a human in one click. Dispositions, transcripts, and lead scores write back to the CRM automatically.

See it live

Book a 30-minute working session at calendly.com/sagar-callsphere/callsphere-llc-meeting and bring a real call flow — we will walk it through the live outbound sales dialer at callsphere.ai/demo and show you exactly where the production wiring sits.