TL;DR — whisper.cpp and llama.cpp are pure-C++ runtimes that run Whisper and Llama-family models on CPU/Metal/CUDA with no Python in the hot path. Glue them with a small Python loop and Piper, and you get a working voice agent on a 2024 MacBook Air with zero outbound traffic.

What you'll build

A single Python process that captures microphone audio in 1-second windows, transcribes with whisper.cpp (base.en quantized), routes the text to a 7B Llama model served by llama-server over its OpenAI-compatible /v1/chat/completions endpoint, and speaks the reply through Piper. Total RAM: ~6 GB. Total network calls: 0.

Prerequisites

1. macOS (Apple Silicon) or Linux with 8 GB+ RAM.
2. cmake, make, and a recent gcc/clang.
3. Python 3.11 with sounddevice, numpy, requests, piper-tts.
4. ~6 GB free disk for models (Whisper base.en + Llama 3.1 8B Q4_K_M + Piper en_US-amy-medium).

Architecture

flowchart LR
 MIC[Microphone] -->|PCM 16kHz| WCPP[whisper.cpp main]
 WCPP -->|text| LOOP[Python loop]
 LOOP -->|HTTP /v1/chat/completions| LSRV[llama-server:8080]
 LSRV -->|text| LOOP
 LOOP -->|text| PIPER[piper-tts]
 PIPER -->|PCM| SPK[Speaker]

Step 1 — Build whisper.cpp and llama.cpp

```bash git clone https://github.com/ggml-org/whisper.cpp && cd whisper.cpp cmake -B build -DGGML_METAL=1 && cmake --build build -j bash./models/download-ggml-model.sh base.en cd.. && git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp cmake -B build -DGGML_METAL=1 && cmake --build build -j ```

On Linux + NVIDIA, swap -DGGML_METAL=1 for -DGGML_CUDA=1. The binaries you care about are build/bin/whisper-cli and build/bin/llama-server.

Step 2 — Start llama-server with an OpenAI-compatible API

```bash

Download a Q4_K_M GGUF of Llama 3.1 8B Instruct from HF first

./build/bin/llama-server \ -m models/llama-3.1-8b-instruct-q4_k_m.gguf \ --host 127.0.0.1 --port 8080 \ --ctx-size 8192 --n-gpu-layers 99 \ --chat-template llama3 ```

llama-server exposes /v1/chat/completions, /v1/completions and /v1/embeddings on the same port — drop-in for the OpenAI Python SDK.

Step 3 — Capture audio and run whisper.cpp on each utterance

```python import sounddevice as sd, numpy as np, subprocess, tempfile, wave, requests SAMPLE_RATE = 16000

def record_until_silence(threshold=0.01, max_seconds=8): frames, silent = [], 0 with sd.InputStream(samplerate=SAMPLE_RATE, channels=1, dtype="int16") as s: while silent < int(SAMPLE_RATE * 0.6) and len(frames) < SAMPLE_RATE * max_seconds: chunk, _ = s.read(1600) frames.append(chunk) rms = np.sqrt(np.mean((chunk.astype(np.float32) / 32768) ** 2)) silent = silent + 1600 if rms < threshold else 0 return np.concatenate(frames)

def transcribe(pcm): f = tempfile.NamedTemporaryFile(suffix=".wav", delete=False).name with wave.open(f, "wb") as w: w.setnchannels(1); w.setsampwidth(2); w.setframerate(SAMPLE_RATE) w.writeframes(pcm.tobytes()) out = subprocess.check_output([ "whisper.cpp/build/bin/whisper-cli", "-m", "whisper.cpp/models/ggml-base.en.bin", "-f", f, "-nt", "-otxt"], text=True) return out.strip() ```

Step 4 — Talk to llama-server and stream the reply through Piper

```python import json SYSTEM = "You are a concise local voice assistant. Reply in 1-2 sentences."

def chat(history, user): history.append({"role": "user", "content": user}) r = requests.post("http://127.0.0.1:8080/v1/chat/completions", json={"model": "local", "messages": [{"role":"system","content":SYSTEM}, *history], "temperature": 0.4, "max_tokens": 160}).json() reply = r["choices"][0]["message"]["content"] history.append({"role": "assistant", "content": reply}) return reply

def speak(text): p = subprocess.Popen(["piper", "--model", "en_US-amy-medium.onnx", "--output-raw"], stdin=subprocess.PIPE, stdout=subprocess.PIPE) raw, _ = p.communicate(text.encode()) sd.play(np.frombuffer(raw, dtype=np.int16), 22050); sd.wait() ```

Step 5 — Glue it into a loop

```python history = [] while True: pcm = record_until_silence() text = transcribe(pcm) if not text or len(text) < 3: continue print("USER:", text) reply = chat(history, text) print("BOT:", reply) speak(reply) ```

Step 6 — Quantize aggressively for older laptops

If you're on an Intel Mac or 8 GB box, swap to Q4_0 or IQ3_M Llama quants and use tiny.en for Whisper. End-to-end latency on M1 base = 1.4 s; on a 2018 Intel i7 = 4.8 s.

Common pitfalls

• Sample-rate mismatch. Whisper expects 16 kHz mono; Piper outputs 22.05 kHz. Don't share the audio device handle.
• llama-server chat template. Llama 3 needs --chat-template llama3; without it, you'll see role-leakage.
• Metal OOM. --n-gpu-layers 99 offloads everything; reduce to 24 if your unified memory pressure spikes.

How CallSphere does this in production

Explore a live demo and compare current plans to find the right fit for your business.

FAQ

Why not just use the OpenAI Realtime API? You will when you go to production. Local is for privacy-sensitive prototyping, on-prem POCs, and offline kiosks.

Can I add tools / function calling? Yes — Llama 3.1 + llama-server supports OpenAI-style tools and tool_choice since b3982.

What's the cheapest TTS that doesn't sound robotic? Piper en_US-amy-medium for English; en_GB-alan-medium is also surprisingly good.

Can this scale? Single-user yes; multi-tenant no — llama-server serializes generations.

Is it really HIPAA-able? Closer than cloud, but you still need a BAA-grade audit trail. CallSphere's Healthcare stack handles that for you.

Sources

• whisper.cpp on GitHub
• llama.cpp server README
• Piper TTS
• llama.cpp 2026 guide