Build a Voice Agent with Krisp Audio Filter and VIVA SDK (2026)

TL;DR — Krisp shipped VIVA (Voice Isolation for Voice Agents) in 2026 — a CPU-only model 3.5x smaller than its predecessor that strips background noise AND secondary voices before audio reaches your STT. Drop it as a pipeline pre-processor and watch WER improve 20-40% on real-world calls.

What you'll build

A LiveKit Agents pipeline with Krisp VIVA inserted between the room input track and Deepgram STT, so the LLM only hears the primary caller — even at a coffee shop or with a TV in the background.

Architecture

flowchart LR
  MIC[Caller mic] --> RM[LiveKit room]
  RM --> KR[Krisp VIVA filter]
  KR -- clean PCM --> STT[Deepgram Nova-3]
  STT --> LLM[GPT-4o]
  LLM --> TTS[ElevenLabs]
  TTS --> RM --> MIC

Step 1 — Get Krisp SDK

Sign up at developers.krisp.ai. You'll get an SDK token + a per-platform binary (libkrisp-audio-sdk.so for Linux, .dylib Mac, .dll Windows, .wasm for browser).

Step 2 — Python bindings

```bash pip install krisp-audio-sdk # internal pip from Krisp export KRISP_TOKEN="your-token" ```

Step 3 — Wrap as an audio processor

```python import numpy as np from krisp_audio_sdk import AudioCleaner, ModelType from livekit.agents import audio

class KrispVAF(audio.AudioProcessor): def init(self): self.cleaner = AudioCleaner( model=ModelType.VIVA_VC_32K, # voice-call optimised sample_rate=16000, frame_size_ms=10, ) async def process(self, frame: audio.AudioFrame) -> audio.AudioFrame: clean = self.cleaner.clean_frame(frame.data) return audio.AudioFrame(data=clean, sample_rate=frame.sample_rate, num_channels=frame.num_channels) ```

Step 4 — Insert into LiveKit pipeline

```python from livekit.agents import AgentSession, RoomInputOptions

session = AgentSession( stt=deepgram.STT(model="nova-3"), llm=openai.LLM(model="gpt-4o"), tts=elevenlabs.TTS(), ) await session.start( room=ctx.room, agent=Concierge(), room_input_options=RoomInputOptions( audio_processors=[KrispVAF()], ), ) ```

Step 5 — Browser fallback (WASM)

```ts import { KrispSDK } from "@krisp.ai/krisp-audio-sdk-wasm";

const krisp = await KrispSDK.create({ authToken: process.env.NEXT_PUBLIC_KRISP_TOKEN!, model: "viva_vc_16k", }); const stream = await navigator.mediaDevices.getUserMedia({ audio: true }); const cleaned = await krisp.process(stream); // returns a MediaStream // pipe cleaned into your WebRTC peer connection or AudioWorklet ```

Step 6 — Pipecat variant

```python from pipecat.audio.filters.krisp_filter import KrispFilter

transport = DailyTransport(..., DailyParams( audio_in_filter=KrispFilter(model="viva_vc_32k"), )) ```

Step 7 — Measure the win

Run a controlled WER test (e.g. LibriSpeech + cafe-noise SNR 10 dB). Typical numbers in 2026: Deepgram Nova-3 alone hits ~14% WER on noisy mixed clips; Nova-3 + VIVA drops to ~9% — a >30% relative reduction.

Pitfalls

• Sample rate: VIVA models are SR-pinned (16k or 32k); resample BEFORE clean_frame.
• CPU budget: VIVA-VC adds ~6-10% single-core CPU per stream; size workers accordingly.
• Frame size: Stick with 10ms — 20ms increases buffer latency 2x for marginal quality gain.
• Browser CORS: WASM build requires Cross-Origin-Embedder-Policy: require-corp in your headers.

How CallSphere does this

CallSphere wraps every inbound call across 6 verticals with VIVA, then feeds 37 agents through 90+ tools and 115+ DB tables. The salon vertical (loud chair-side noise) saw a 33% WER reduction. $149/$499/$1,499 · 14-day trial · 22% affiliate.

FAQ

Cloud or local? Krisp processes locally — no audio leaves the worker, so HIPAA/PII stays sealed.

License model? Per-minute via SDK token; volume tiers down to ~$0.001/min at scale.

Mobile? iOS + Android binaries ship with the same API surface.

Compatible with Deepgram/AssemblyAI/Soniox? Yes — VIVA is a pre-processor, totally vendor-neutral.

Sources

• Krisp Developers - Real-Time AI Voice SDK - https://krisp.ai/developers/
• Krisp SDK Docs - https://sdk-docs.krisp.ai/
• Krisp Blog - 3.5x Smaller Voice Isolation Model - https://krisp.ai/blog/small-voice-isolation-model/
• Krisp SDK Docs - Twilio Voice Integration - https://sdk-docs.krisp.ai/docs/twilio-voice

## How this plays out in production If you are taking the ideas in *Build a Voice Agent with Krisp Audio Filter and VIVA SDK (2026)* and putting them in front of real customers, the constraint that decides everything is ASR error rates on long-tail entities (drug names, street names, SKUs) and the post-call pipeline that must reconcile what was actually heard. Treat this as a voice-first system from the first prompt: the agent's persona, its tool surface, and its escalation rules all flow from that single decision. Teams that ship fast tend to instrument the loop end-to-end before they tune any single component, because the bottleneck is rarely where intuition puts it. ## Voice agent architecture, end to end A production-grade voice stack at CallSphere stitches Twilio Programmable Voice (PSTN ingress, TwiML, bidirectional Media Streams) to a realtime reasoning layer — typically OpenAI Realtime or ElevenLabs Conversational AI — with sub-second response as a hard SLO. Anything north of one second of perceived silence and callers either repeat themselves or hang up; that single number drives the whole architecture. Server-side VAD with proper barge-in support is non-negotiable, otherwise the agent talks over the caller and the conversation collapses. Streaming TTS with phoneme-aligned interruption keeps the cadence natural even when the user changes their mind mid-sentence. Post-call, every transcript is run through a structured pipeline: sentiment, intent classification, lead score, escalation flag, and a normalized slot extraction (name, callback number, reason, urgency). For healthcare workloads, the BAA-covered storage path, audit logs, encryption-at-rest, and PHI-safe transcript redaction are wired in from day one, not bolted on at compliance review. The end state is a system where every call produces a row of structured data, not just a recording. ## FAQ **What does this mean for a voice agent the way *Build a Voice Agent with Krisp Audio Filter and VIVA SDK (2026)* describes?** Treat the architecture in this post as a starting point and instrument it before you tune it. The metrics that matter most early on are end-to-end latency (target < 1s for voice, < 3s for chat), barge-in correctness, tool-call success rate, and post-conversation lead score distribution. Optimize whatever the data flags as the bottleneck, not whatever feels slowest in your head. **Why does this matter for voice agent deployments at scale?** The two failure modes that bite hardest are silent context loss across multi-turn handoffs and tool calls that succeed in dev but get rate-limited in production. Both are solvable with a proper agent backplane that pins state to a session ID, retries with backoff, and writes every tool invocation to an audit log you can replay. **How does the salon stack (GlamBook) keep bookings clean across stylists and services?** GlamBook runs 4 agents that handle booking, rescheduling, fuzzy service-name matching, and confirmations. Every appointment gets a deterministic reference like GB-YYYYMMDD-### so the salon, the customer, and the agent all reference the same object across SMS, email, and voice. ## See it live Book a 30-minute working session at [calendly.com/sagar-callsphere/new-meeting](https://calendly.com/sagar-callsphere/new-meeting) and bring a real call flow — we will walk it through the live salon booking agent (GlamBook) at [salon.callsphere.tech](https://salon.callsphere.tech) and show you exactly where the production wiring sits.