Voice Agent Testing and Quality Assurance Strategies

Why Voice Agent Testing Is Different

Testing a chat agent is straightforward: send text in, check text out. Testing a voice agent involves multiple layers that can each fail independently — speech-to-text accuracy, natural language understanding, tool execution, response generation, and text-to-speech quality. A bug at any layer degrades the user experience, and the layers interact in ways that are hard to predict from unit tests alone.

Voice agents also have timing-sensitive behaviors (VAD, turn-taking, barge-in) that require audio-level testing, not just text-level testing. This post covers a complete QA strategy from unit tests through end-to-end conversation simulation.

Layer 1: Speech-to-Text Accuracy Testing

The STT layer converts user audio into text. If the transcription is wrong, everything downstream fails. Test STT accuracy systematically:

flowchart LR
    PR(["PR opened"])
    UNIT["Unit tests"]
    EVAL["Eval harness<br/>PromptFoo or Braintrust"]
    GOLD[("Golden set<br/>200 tagged cases")]
    JUDGE["LLM as judge<br/>plus regex graders"]
    SCORE["Aggregate score<br/>and per slice"]
    GATE{"Score regress<br/>more than 2 percent?"}
    BLOCK(["Block merge"])
    MERGE(["Merge to main"])
    PR --> UNIT --> EVAL --> GOLD --> JUDGE --> SCORE --> GATE
    GATE -->|Yes| BLOCK
    GATE -->|No| MERGE
    style EVAL fill:#4f46e5,stroke:#4338ca,color:#fff
    style GATE fill:#f59e0b,stroke:#d97706,color:#1f2937
    style BLOCK fill:#dc2626,stroke:#b91c1c,color:#fff
    style MERGE fill:#059669,stroke:#047857,color:#fff

from dataclasses import dataclass
from typing import Optional
import json

@dataclass
class STTTestCase:
    audio_file: str
    expected_transcript: str
    language: str = "en"
    accent: Optional[str] = None
    noise_level: Optional[str] = None  # "quiet", "moderate", "noisy"
    description: str = ""

STT_TEST_SUITE = [
    STTTestCase(
        audio_file="tests/audio/booking_clean.wav",
        expected_transcript="I would like to book an appointment for tomorrow at 2 PM",
        noise_level="quiet",
        description="Clean booking request",
    ),
    STTTestCase(
        audio_file="tests/audio/booking_noisy.wav",
        expected_transcript="I would like to book an appointment for tomorrow at 2 PM",
        noise_level="noisy",
        description="Same request with background noise",
    ),
    STTTestCase(
        audio_file="tests/audio/phone_number.wav",
        expected_transcript="My phone number is 555-012-3456",
        noise_level="quiet",
        description="Phone number dictation — tests digit accuracy",
    ),
    STTTestCase(
        audio_file="tests/audio/name_spelling.wav",
        expected_transcript="My name is Krishnamurthy, K-R-I-S-H-N-A-M-U-R-T-H-Y",
        noise_level="quiet",
        accent="south-asian",
        description="Name with spelling — tests uncommon word handling",
    ),
]

Measuring STT Quality

Use Word Error Rate (WER) as the primary metric:

def word_error_rate(reference: str, hypothesis: str) -> float:
    """Calculate Word Error Rate between reference and hypothesis transcripts."""
    ref_words = reference.lower().split()
    hyp_words = hypothesis.lower().split()

    # Dynamic programming for edit distance
    d = [[0] * (len(hyp_words) + 1) for _ in range(len(ref_words) + 1)]

    for i in range(len(ref_words) + 1):
        d[i][0] = i
    for j in range(len(hyp_words) + 1):
        d[0][j] = j

    for i in range(1, len(ref_words) + 1):
        for j in range(1, len(hyp_words) + 1):
            if ref_words[i - 1] == hyp_words[j - 1]:
                d[i][j] = d[i - 1][j - 1]
            else:
                d[i][j] = min(
                    d[i - 1][j] + 1,      # deletion
                    d[i][j - 1] + 1,      # insertion
                    d[i - 1][j - 1] + 1,  # substitution
                )

    return d[len(ref_words)][len(hyp_words)] / max(len(ref_words), 1)

async def run_stt_tests(test_cases: list[STTTestCase]) -> dict:
    """Run the STT test suite and return aggregate metrics."""
    results = []
    for case in test_cases:
        transcript = await transcribe_audio(case.audio_file)
        wer = word_error_rate(case.expected_transcript, transcript)
        results.append({
            "description": case.description,
            "expected": case.expected_transcript,
            "actual": transcript,
            "wer": wer,
            "noise_level": case.noise_level,
            "passed": wer < 0.1,  # 10% WER threshold
        })

    total = len(results)
    passed = sum(1 for r in results if r["passed"])
    avg_wer = sum(r["wer"] for r in results) / max(total, 1)

    return {
        "total": total,
        "passed": passed,
        "failed": total - passed,
        "average_wer": round(avg_wer, 4),
        "details": results,
    }

Set WER thresholds by category: clean audio should be under 5%, noisy audio under 15%, and accented speech under 10%. Track these over time to catch regressions.

Layer 2: Conversation Logic Testing

Once you trust the STT layer, test the agent's conversational logic using text inputs (bypassing audio):

import pytest
from agents import Agent, Runner

@pytest.fixture
def booking_agent():
    return Agent(
        name="BookingAgent",
        instructions="You are a medical appointment booking assistant...",
        tools=[check_availability, book_appointment],
    )

@pytest.mark.asyncio
async def test_booking_happy_path(booking_agent):
    """Test a complete booking flow with all required information."""
    result = await Runner.run(
        booking_agent,
        "I need to see Dr. Smith tomorrow at 2pm for a cleaning. "
        "My name is Jane Doe and my number is 555-0199.",
    )
    output = result.final_output.lower()

    # Agent should have called check_availability and book_appointment
    tool_names = [
        item.raw_item.name
        for item in result.new_items
        if hasattr(item, "raw_item") and hasattr(item.raw_item, "name")
    ]
    assert "check_availability" in tool_names or "book_appointment" in tool_names
    assert "confirmation" in output or "booked" in output

@pytest.mark.asyncio
async def test_missing_information_prompts(booking_agent):
    """Test that the agent asks for missing required fields."""
    result = await Runner.run(
        booking_agent,
        "I want to book an appointment.",
    )
    output = result.final_output.lower()

    # Agent should ask for details, not attempt to book
    assert any(
        word in output
        for word in ["which", "what", "when", "who", "provider", "date", "time"]
    )

@pytest.mark.asyncio
async def test_unavailable_slot_handling(booking_agent):
    """Test graceful handling when requested slot is unavailable."""
    result = await Runner.run(
        booking_agent,
        "Book me with Dr. Smith on December 25th at 3am for surgery. "
        "Name: Test User, phone: 555-0000.",
    )
    output = result.final_output.lower()

    # Should suggest alternatives, not crash
    assert "available" in output or "alternative" in output or "another" in output

Layer 3: TTS Quality Evaluation

Text-to-speech quality is subjective, but you can measure objective aspects programmatically. The most powerful automated technique is the round-trip test: generate TTS audio from text, then transcribe it back with STT and measure the Word Error Rate. If the STT cannot understand the TTS output, neither can most humans. Target 95%+ round-trip intelligibility.

Key metrics to track include: intelligibility percentage (round-trip WER), speaking pace in words per minute (target 140-170 WPM for English), pause accuracy (do pauses align with punctuation?), and pronunciation errors on domain-specific terms like medical terminology or proper nouns.

Layer 4: End-to-End Conversation Simulation

The most comprehensive test simulates full voice conversations. Use a test harness that generates audio, sends it to the voice agent, and evaluates the response:

from dataclasses import dataclass, field

@dataclass
class ConversationTurn:
    user_text: str
    expected_intent: str
    expected_tools: list = field(default_factory=list)
    expected_keywords: list = field(default_factory=list)
    max_response_time_ms: int = 3000

@dataclass
class ConversationScenario:
    name: str
    description: str
    turns: list[ConversationTurn] = field(default_factory=list)

BOOKING_SCENARIO = ConversationScenario(
    name="complete_booking_flow",
    description="Full appointment booking from greeting to confirmation",
    turns=[
        ConversationTurn(
            user_text="Hi, I need to make an appointment.",
            expected_intent="greeting_and_request",
            expected_keywords=["help", "appointment", "provider", "when"],
        ),
        ConversationTurn(
            user_text="I need to see a dentist, preferably Dr. Smith.",
            expected_intent="provider_selection",
            expected_tools=["check_availability"],
            expected_keywords=["available", "smith"],
        ),
        ConversationTurn(
            user_text="Tomorrow at 2pm works.",
            expected_intent="time_selection",
            expected_keywords=["2", "pm", "confirm"],
        ),
        ConversationTurn(
            user_text="Yes, my name is Jane Doe and my number is 555-0199.",
            expected_intent="provide_details",
            expected_tools=["book_appointment"],
            expected_keywords=["booked", "confirmation"],
        ),
        ConversationTurn(
            user_text="Can you text me the confirmation?",
            expected_intent="sms_request",
            expected_tools=["send_sms_confirmation"],
            expected_keywords=["sent", "text", "sms"],
        ),
    ],
)

Running Conversation Simulations

The simulation runner iterates through each turn, sends the user text to the agent, and evaluates the response against expected keywords, tool calls, and response time thresholds. Each turn result includes whether keywords were found, which tools were called, and whether the response time was within the allowed maximum. The scenario passes only if every turn passes.

Store conversation history across turns so the agent has full context, just like a real voice session. After each turn, append both the user message and agent response to the history array before proceeding to the next turn.

Regression Monitoring

Voice agent quality can regress silently — a model update changes phrasing, a tool API changes its response format, or a VAD threshold drift causes more interruptions. Set up continuous monitoring:

Define a QualityBaseline dataclass with fields for STT WER, average response time, tool success rate, conversation completion rate, interruption rate, and user satisfaction score. Then write a check_regression function that compares current metrics against the baseline using threshold multipliers: flag STT WER if it exceeds 1.2x baseline, response time if it exceeds 1.3x, and tool/completion rates if they drop below 0.9x baseline. Return a list of alert strings for any regressions detected.

Putting It All Together: CI Pipeline

Integrate voice agent tests into your CI/CD pipeline with tiered execution:

1. On every commit: Run conversation logic tests (text-only, fast, no audio)
2. On every PR: Run STT accuracy tests with cached audio samples
3. Nightly: Run full end-to-end conversation simulations with audio generation
4. Weekly: Run TTS quality evaluation and regression checks against baseline
5. Monthly: Refresh audio test data with new recordings and noise profiles

Voice agent QA is an investment that compounds over time. Every test case you add catches future regressions before users encounter them. Start with conversation logic tests (cheapest to write and run), then layer on STT and TTS tests as your agent matures. The goal is not perfection on day one — it is a ratchet that only moves in the direction of better quality.