Build a Voice Agent with Krisp Audio Filter and VIVA SDK (2026)

TL;DR — Krisp shipped VIVA (Voice Isolation for Voice Agents) in 2026 — a CPU-only model 3.5x smaller than its predecessor that strips background noise AND secondary voices before audio reaches your STT. Drop it as a pipeline pre-processor and watch WER improve 20-40% on real-world calls.

What you'll build

A LiveKit Agents pipeline with Krisp VIVA inserted between the room input track and Deepgram STT, so the LLM only hears the primary caller — even at a coffee shop or with a TV in the background.

Architecture

flowchart LR
  MIC[Caller mic] --> RM[LiveKit room]
  RM --> KR[Krisp VIVA filter]
  KR -- clean PCM --> STT[Deepgram Nova-3]
  STT --> LLM[GPT-4o]
  LLM --> TTS[ElevenLabs]
  TTS --> RM --> MIC

Step 1 — Get Krisp SDK

Sign up at developers.krisp.ai. You'll get an SDK token + a per-platform binary (libkrisp-audio-sdk.so for Linux, .dylib Mac, .dll Windows, .wasm for browser).

Step 2 — Python bindings

```bash pip install krisp-audio-sdk # internal pip from Krisp export KRISP_TOKEN="your-token" ```

Step 3 — Wrap as an audio processor

```python import numpy as np from krisp_audio_sdk import AudioCleaner, ModelType from livekit.agents import audio

class KrispVAF(audio.AudioProcessor): def init(self): self.cleaner = AudioCleaner( model=ModelType.VIVA_VC_32K, # voice-call optimised sample_rate=16000, frame_size_ms=10, ) async def process(self, frame: audio.AudioFrame) -> audio.AudioFrame: clean = self.cleaner.clean_frame(frame.data) return audio.AudioFrame(data=clean, sample_rate=frame.sample_rate, num_channels=frame.num_channels) ```

Step 4 — Insert into LiveKit pipeline

```python from livekit.agents import AgentSession, RoomInputOptions

session = AgentSession( stt=deepgram.STT(model="nova-3"), llm=openai.LLM(model="gpt-4o"), tts=elevenlabs.TTS(), ) await session.start( room=ctx.room, agent=Concierge(), room_input_options=RoomInputOptions( audio_processors=[KrispVAF()], ), ) ```

Step 5 — Browser fallback (WASM)

```ts import { KrispSDK } from "@krisp.ai/krisp-audio-sdk-wasm";

const krisp = await KrispSDK.create({ authToken: process.env.NEXT_PUBLIC_KRISP_TOKEN!, model: "viva_vc_16k", }); const stream = await navigator.mediaDevices.getUserMedia({ audio: true }); const cleaned = await krisp.process(stream); // returns a MediaStream // pipe cleaned into your WebRTC peer connection or AudioWorklet ```

Step 6 — Pipecat variant

```python from pipecat.audio.filters.krisp_filter import KrispFilter

transport = DailyTransport(..., DailyParams( audio_in_filter=KrispFilter(model="viva_vc_32k"), )) ```

Step 7 — Measure the win

Run a controlled WER test (e.g. LibriSpeech + cafe-noise SNR 10 dB). Typical numbers in 2026: Deepgram Nova-3 alone hits ~14% WER on noisy mixed clips; Nova-3 + VIVA drops to ~9% — a >30% relative reduction.

Pitfalls

• Sample rate: VIVA models are SR-pinned (16k or 32k); resample BEFORE clean_frame.
• CPU budget: VIVA-VC adds ~6-10% single-core CPU per stream; size workers accordingly.
• Frame size: Stick with 10ms — 20ms increases buffer latency 2x for marginal quality gain.
• Browser CORS: WASM build requires Cross-Origin-Embedder-Policy: require-corp in your headers.

How CallSphere does this

Explore a live demo and compare current plans to find the right fit for your business.

FAQ

Cloud or local? Krisp processes locally — no audio leaves the worker, so HIPAA/PII stays sealed.

License model? Per-minute via SDK token; volume tiers down to ~$0.001/min at scale.

Mobile? iOS + Android binaries ship with the same API surface.

Compatible with Deepgram/AssemblyAI/Soniox? Yes — VIVA is a pre-processor, totally vendor-neutral.

Sources

• Krisp Developers - Real-Time AI Voice SDK - https://krisp.ai/developers/
• Krisp SDK Docs - https://sdk-docs.krisp.ai/
• Krisp Blog - 3.5x Smaller Voice Isolation Model - https://krisp.ai/blog/small-voice-isolation-model/
• Krisp SDK Docs - Twilio Voice Integration - https://sdk-docs.krisp.ai/docs/twilio-voice

How this plays out in production

If you are taking the ideas in Build a Voice Agent with Krisp Audio Filter and VIVA SDK (2026) and putting them in front of real customers, the constraint that decides everything is ASR error rates on long-tail entities (drug names, street names, SKUs) and the post-call pipeline that must reconcile what was actually heard. Treat this as a voice-first system from the first prompt: the agent's persona, its tool surface, and its escalation rules all flow from that single decision. Teams that ship fast tend to instrument the loop end-to-end before they tune any single component, because the bottleneck is rarely where intuition puts it.

Voice agent architecture, end to end

A production-grade voice stack at CallSphere stitches Twilio Programmable Voice (PSTN ingress, TwiML, bidirectional Media Streams) to a realtime reasoning layer — typically OpenAI Realtime or ElevenLabs Conversational AI — with sub-second response as a hard SLO. Anything north of one second of perceived silence and callers either repeat themselves or hang up; that single number drives the whole architecture. Server-side VAD with proper barge-in support is non-negotiable, otherwise the agent talks over the caller and the conversation collapses. Streaming TTS with phoneme-aligned interruption keeps the cadence natural even when the user changes their mind mid-sentence. Post-call, every transcript is run through a structured pipeline: sentiment, intent classification, lead score, escalation flag, and a normalized slot extraction (name, callback number, reason, urgency). For healthcare workloads, the BAA-covered storage path, audit logs, encryption-at-rest, and PHI-safe transcript redaction are wired in from day one, not bolted on at compliance review. The end state is a system where every call produces a row of structured data, not just a recording.

FAQ

What does this mean for a voice agent the way Build a Voice Agent with Krisp Audio Filter and VIVA SDK (2026) describes?

Treat the architecture in this post as a starting point and instrument it before you tune it. The metrics that matter most early on are end-to-end latency (target < 1s for voice, < 3s for chat), barge-in correctness, tool-call success rate, and post-conversation lead score distribution. Optimize whatever the data flags as the bottleneck, not whatever feels slowest in your head.

Why does this matter for voice agent deployments at scale?

The two failure modes that bite hardest are silent context loss across multi-turn handoffs and tool calls that succeed in dev but get rate-limited in production. Both are solvable with a proper agent backplane that pins state to a session ID, retries with backoff, and writes every tool invocation to an audit log you can replay.

How does the salon stack (GlamBook) keep bookings clean across stylists and services?

GlamBook runs 4 agents that handle booking, rescheduling, fuzzy service-name matching, and confirmations. Every appointment gets a deterministic reference like GB-YYYYMMDD-### so the salon, the customer, and the agent all reference the same object across SMS, email, and voice.

See it live

Book a 30-minute working session at calendly.com/sagar-callsphere/callsphere-llc-meeting and bring a real call flow — we will walk it through the live salon booking agent (GlamBook) at salon.callsphere.tech and show you exactly where the production wiring sits.