Voice Agent Testing and Quality Assurance Strategies

Why Voice Agent Testing Is Different

Testing a chat agent is straightforward: send text in, check text out. Testing a voice agent involves multiple layers that can each fail independently — speech-to-text accuracy, natural language understanding, tool execution, response generation, and text-to-speech quality. A bug at any layer degrades the user experience, and the layers interact in ways that are hard to predict from unit tests alone.

Voice agents also have timing-sensitive behaviors (VAD, turn-taking, barge-in) that require audio-level testing, not just text-level testing. This post covers a complete QA strategy from unit tests through end-to-end conversation simulation.

Layer 1: Speech-to-Text Accuracy Testing

The STT layer converts user audio into text. If the transcription is wrong, everything downstream fails. Test STT accuracy systematically:

flowchart TD
    START["Voice Agent Testing and Quality Assurance Strateg…"] --> A
    A["Why Voice Agent Testing Is Different"]
    A --> B
    B["Layer 1: Speech-to-Text Accuracy Testing"]
    B --> C
    C["Layer 2: Conversation Logic Testing"]
    C --> D
    D["Layer 3: TTS Quality Evaluation"]
    D --> E
    E["Layer 4: End-to-End Conversation Simula…"]
    E --> F
    F["Regression Monitoring"]
    F --> G
    G["Putting It All Together: CI Pipeline"]
    G --> DONE["Key Takeaways"]
    style START fill:#4f46e5,stroke:#4338ca,color:#fff
    style DONE fill:#059669,stroke:#047857,color:#fff

from dataclasses import dataclass
from typing import Optional
import json

@dataclass
class STTTestCase:
    audio_file: str
    expected_transcript: str
    language: str = "en"
    accent: Optional[str] = None
    noise_level: Optional[str] = None  # "quiet", "moderate", "noisy"
    description: str = ""

STT_TEST_SUITE = [
    STTTestCase(
        audio_file="tests/audio/booking_clean.wav",
        expected_transcript="I would like to book an appointment for tomorrow at 2 PM",
        noise_level="quiet",
        description="Clean booking request",
    ),
    STTTestCase(
        audio_file="tests/audio/booking_noisy.wav",
        expected_transcript="I would like to book an appointment for tomorrow at 2 PM",
        noise_level="noisy",
        description="Same request with background noise",
    ),
    STTTestCase(
        audio_file="tests/audio/phone_number.wav",
        expected_transcript="My phone number is 555-012-3456",
        noise_level="quiet",
        description="Phone number dictation — tests digit accuracy",
    ),
    STTTestCase(
        audio_file="tests/audio/name_spelling.wav",
        expected_transcript="My name is Krishnamurthy, K-R-I-S-H-N-A-M-U-R-T-H-Y",
        noise_level="quiet",
        accent="south-asian",
        description="Name with spelling — tests uncommon word handling",
    ),
]

Measuring STT Quality

Use Word Error Rate (WER) as the primary metric:

def word_error_rate(reference: str, hypothesis: str) -> float:
    """Calculate Word Error Rate between reference and hypothesis transcripts."""
    ref_words = reference.lower().split()
    hyp_words = hypothesis.lower().split()

    # Dynamic programming for edit distance
    d = [[0] * (len(hyp_words) + 1) for _ in range(len(ref_words) + 1)]

    for i in range(len(ref_words) + 1):
        d[i][0] = i
    for j in range(len(hyp_words) + 1):
        d[0][j] = j

    for i in range(1, len(ref_words) + 1):
        for j in range(1, len(hyp_words) + 1):
            if ref_words[i - 1] == hyp_words[j - 1]:
                d[i][j] = d[i - 1][j - 1]
            else:
                d[i][j] = min(
                    d[i - 1][j] + 1,      # deletion
                    d[i][j - 1] + 1,      # insertion
                    d[i - 1][j - 1] + 1,  # substitution
                )

    return d[len(ref_words)][len(hyp_words)] / max(len(ref_words), 1)

async def run_stt_tests(test_cases: list[STTTestCase]) -> dict:
    """Run the STT test suite and return aggregate metrics."""
    results = []
    for case in test_cases:
        transcript = await transcribe_audio(case.audio_file)
        wer = word_error_rate(case.expected_transcript, transcript)
        results.append({
            "description": case.description,
            "expected": case.expected_transcript,
            "actual": transcript,
            "wer": wer,
            "noise_level": case.noise_level,
            "passed": wer < 0.1,  # 10% WER threshold
        })

    total = len(results)
    passed = sum(1 for r in results if r["passed"])
    avg_wer = sum(r["wer"] for r in results) / max(total, 1)

    return {
        "total": total,
        "passed": passed,
        "failed": total - passed,
        "average_wer": round(avg_wer, 4),
        "details": results,
    }

Set WER thresholds by category: clean audio should be under 5%, noisy audio under 15%, and accented speech under 10%. Track these over time to catch regressions.

Layer 2: Conversation Logic Testing

Once you trust the STT layer, test the agent's conversational logic using text inputs (bypassing audio):

import pytest
from agents import Agent, Runner

@pytest.fixture
def booking_agent():
    return Agent(
        name="BookingAgent",
        instructions="You are a medical appointment booking assistant...",
        tools=[check_availability, book_appointment],
    )

@pytest.mark.asyncio
async def test_booking_happy_path(booking_agent):
    """Test a complete booking flow with all required information."""
    result = await Runner.run(
        booking_agent,
        "I need to see Dr. Smith tomorrow at 2pm for a cleaning. "
        "My name is Jane Doe and my number is 555-0199.",
    )
    output = result.final_output.lower()

    # Agent should have called check_availability and book_appointment
    tool_names = [
        item.raw_item.name
        for item in result.new_items
        if hasattr(item, "raw_item") and hasattr(item.raw_item, "name")
    ]
    assert "check_availability" in tool_names or "book_appointment" in tool_names
    assert "confirmation" in output or "booked" in output

@pytest.mark.asyncio
async def test_missing_information_prompts(booking_agent):
    """Test that the agent asks for missing required fields."""
    result = await Runner.run(
        booking_agent,
        "I want to book an appointment.",
    )
    output = result.final_output.lower()

    # Agent should ask for details, not attempt to book
    assert any(
        word in output
        for word in ["which", "what", "when", "who", "provider", "date", "time"]
    )

@pytest.mark.asyncio
async def test_unavailable_slot_handling(booking_agent):
    """Test graceful handling when requested slot is unavailable."""
    result = await Runner.run(
        booking_agent,
        "Book me with Dr. Smith on December 25th at 3am for surgery. "
        "Name: Test User, phone: 555-0000.",
    )
    output = result.final_output.lower()

    # Should suggest alternatives, not crash
    assert "available" in output or "alternative" in output or "another" in output

Layer 3: TTS Quality Evaluation

Text-to-speech quality is subjective, but you can measure objective aspects programmatically. The most powerful automated technique is the round-trip test: generate TTS audio from text, then transcribe it back with STT and measure the Word Error Rate. If the STT cannot understand the TTS output, neither can most humans. Target 95%+ round-trip intelligibility.

flowchart TD
    ROOT["Voice Agent Testing and Quality Assurance St…"] 
    ROOT --> P0["Layer 1: Speech-to-Text Accuracy Testing"]
    P0 --> P0C0["Measuring STT Quality"]
    ROOT --> P1["Layer 4: End-to-End Conversation Simula…"]
    P1 --> P1C0["Running Conversation Simulations"]
    style ROOT fill:#4f46e5,stroke:#4338ca,color:#fff
    style P0 fill:#e0e7ff,stroke:#6366f1,color:#1e293b
    style P1 fill:#e0e7ff,stroke:#6366f1,color:#1e293b

Key metrics to track include: intelligibility percentage (round-trip WER), speaking pace in words per minute (target 140-170 WPM for English), pause accuracy (do pauses align with punctuation?), and pronunciation errors on domain-specific terms like medical terminology or proper nouns.

Layer 4: End-to-End Conversation Simulation

The most comprehensive test simulates full voice conversations. Use a test harness that generates audio, sends it to the voice agent, and evaluates the response:

flowchart TD
    CENTER(("Core Concepts"))
    CENTER --> N0["On every commit: Run conversation logic…"]
    CENTER --> N1["On every PR: Run STT accuracy tests wit…"]
    CENTER --> N2["Nightly: Run full end-to-end conversati…"]
    CENTER --> N3["Weekly: Run TTS quality evaluation and …"]
    CENTER --> N4["Monthly: Refresh audio test data with n…"]
    style CENTER fill:#4f46e5,stroke:#4338ca,color:#fff

from dataclasses import dataclass, field

@dataclass
class ConversationTurn:
    user_text: str
    expected_intent: str
    expected_tools: list = field(default_factory=list)
    expected_keywords: list = field(default_factory=list)
    max_response_time_ms: int = 3000

@dataclass
class ConversationScenario:
    name: str
    description: str
    turns: list[ConversationTurn] = field(default_factory=list)

BOOKING_SCENARIO = ConversationScenario(
    name="complete_booking_flow",
    description="Full appointment booking from greeting to confirmation",
    turns=[
        ConversationTurn(
            user_text="Hi, I need to make an appointment.",
            expected_intent="greeting_and_request",
            expected_keywords=["help", "appointment", "provider", "when"],
        ),
        ConversationTurn(
            user_text="I need to see a dentist, preferably Dr. Smith.",
            expected_intent="provider_selection",
            expected_tools=["check_availability"],
            expected_keywords=["available", "smith"],
        ),
        ConversationTurn(
            user_text="Tomorrow at 2pm works.",
            expected_intent="time_selection",
            expected_keywords=["2", "pm", "confirm"],
        ),
        ConversationTurn(
            user_text="Yes, my name is Jane Doe and my number is 555-0199.",
            expected_intent="provide_details",
            expected_tools=["book_appointment"],
            expected_keywords=["booked", "confirmation"],
        ),
        ConversationTurn(
            user_text="Can you text me the confirmation?",
            expected_intent="sms_request",
            expected_tools=["send_sms_confirmation"],
            expected_keywords=["sent", "text", "sms"],
        ),
    ],
)

Running Conversation Simulations

The simulation runner iterates through each turn, sends the user text to the agent, and evaluates the response against expected keywords, tool calls, and response time thresholds. Each turn result includes whether keywords were found, which tools were called, and whether the response time was within the allowed maximum. The scenario passes only if every turn passes.

Store conversation history across turns so the agent has full context, just like a real voice session. After each turn, append both the user message and agent response to the history array before proceeding to the next turn.

Regression Monitoring

Voice agent quality can regress silently — a model update changes phrasing, a tool API changes its response format, or a VAD threshold drift causes more interruptions. Set up continuous monitoring:

Define a QualityBaseline dataclass with fields for STT WER, average response time, tool success rate, conversation completion rate, interruption rate, and user satisfaction score. Then write a check_regression function that compares current metrics against the baseline using threshold multipliers: flag STT WER if it exceeds 1.2x baseline, response time if it exceeds 1.3x, and tool/completion rates if they drop below 0.9x baseline. Return a list of alert strings for any regressions detected.

Putting It All Together: CI Pipeline

Integrate voice agent tests into your CI/CD pipeline with tiered execution:

1. On every commit: Run conversation logic tests (text-only, fast, no audio)
2. On every PR: Run STT accuracy tests with cached audio samples
3. Nightly: Run full end-to-end conversation simulations with audio generation
4. Weekly: Run TTS quality evaluation and regression checks against baseline
5. Monthly: Refresh audio test data with new recordings and noise profiles

Voice agent QA is an investment that compounds over time. Every test case you add catches future regressions before users encounter them. Start with conversation logic tests (cheapest to write and run), then layer on STT and TTS tests as your agent matures. The goal is not perfection on day one — it is a ratchet that only moves in the direction of better quality.