LLM Response Quality Monitoring: Detecting Degradation in Production

The Silent Problem of Quality Degradation

LLM quality can degrade without any errors being thrown. A model provider pushes a silent update that changes behavior. Your prompt works differently after hitting a new context window boundary. A data pipeline feeds stale information to your retrieval system. The agent still returns HTTP 200 with well-formed JSON, but the answers are subtly worse — less accurate, more verbose, or missing key details.

Unlike latency spikes or error rate increases, quality degradation does not set off traditional alarms. By the time users complain, hundreds or thousands of conversations have already been affected. Automated quality monitoring closes this gap by scoring a sample of production responses and alerting when scores drift below acceptable thresholds.

Defining Quality Metrics

Quality is multidimensional. Define metrics that capture the dimensions most important to your use case.

flowchart LR
    PR(["PR opened"])
    UNIT["Unit tests"]
    EVAL["Eval harness<br/>PromptFoo or Braintrust"]
    GOLD[("Golden set<br/>200 tagged cases")]
    JUDGE["LLM as judge<br/>plus regex graders"]
    SCORE["Aggregate score<br/>and per slice"]
    GATE{"Score regress<br/>more than 2 percent?"}
    BLOCK(["Block merge"])
    MERGE(["Merge to main"])
    PR --> UNIT --> EVAL --> GOLD --> JUDGE --> SCORE --> GATE
    GATE -->|Yes| BLOCK
    GATE -->|No| MERGE
    style EVAL fill:#4f46e5,stroke:#4338ca,color:#fff
    style GATE fill:#f59e0b,stroke:#d97706,color:#1f2937
    style BLOCK fill:#dc2626,stroke:#b91c1c,color:#fff
    style MERGE fill:#059669,stroke:#047857,color:#fff

from dataclasses import dataclass
from enum import Enum

class QualityDimension(Enum):
    RELEVANCE = "relevance"         # Does the response address the question?
    ACCURACY = "accuracy"           # Are the facts correct?
    COMPLETENESS = "completeness"   # Does it cover all aspects of the question?
    CONCISENESS = "conciseness"     # Is it appropriately brief?
    SAFETY = "safety"               # Does it avoid harmful content?
    INSTRUCTION_FOLLOWING = "instruction_following"  # Does it follow the system prompt?

@dataclass
class QualityScore:
    conversation_id: str
    dimension: QualityDimension
    score: float  # 0.0 to 1.0
    explanation: str
    evaluator: str  # "llm-judge", "heuristic", "human"

Building an Automated Scoring Pipeline

Use a separate LLM as a judge to score production responses. This is cost-effective for sampling and scales better than human evaluation.

import json

JUDGE_PROMPT = """You are evaluating the quality of an AI assistant's response.

User question: {question}
Assistant response: {response}

Score each dimension from 0.0 (terrible) to 1.0 (excellent):
- relevance: Does the response directly address the user's question?
- accuracy: Are the claims factually correct?
- completeness: Are all important aspects covered?
- conciseness: Is the response appropriately concise?

Return JSON only:
{{"relevance": 0.0, "accuracy": 0.0, "completeness": 0.0, "conciseness": 0.0, "explanation": "brief reasoning"}}
"""

async def score_response(
    question: str,
    response: str,
    conversation_id: str,
) -> list[QualityScore]:
    judge_response = await judge_client.chat.completions.create(
        model="gpt-4o-mini",  # Use a cheaper model as judge
        messages=[
            {"role": "user", "content": JUDGE_PROMPT.format(
                question=question, response=response
            )},
        ],
        response_format={"type": "json_object"},
        temperature=0.0,
    )

    scores_dict = json.loads(judge_response.choices[0].message.content)
    explanation = scores_dict.pop("explanation", "")

    return [
        QualityScore(
            conversation_id=conversation_id,
            dimension=QualityDimension(dim),
            score=score,
            explanation=explanation,
            evaluator="llm-judge",
        )
        for dim, score in scores_dict.items()
        if dim in QualityDimension._value2member_map_
    ]

Heuristic Quality Checks

Not every quality signal needs an LLM judge. Fast heuristic checks catch obvious problems at zero cost.

import re

def heuristic_quality_checks(response: str, question: str) -> dict[str, float]:
    checks = {}

    # Check for refusals
    refusal_phrases = ["i cannot", "i'm unable", "as an ai", "i don't have access"]
    checks["non_refusal"] = 0.0 if any(p in response.lower() for p in refusal_phrases) else 1.0

    # Check for excessive length (more than 5x the question length is suspicious)
    length_ratio = len(response) / max(len(question), 1)
    checks["length_appropriate"] = 1.0 if length_ratio < 10 else max(0.0, 1.0 - (length_ratio - 10) / 20)

    # Check for hallucination markers
    hedging = ["i think", "i believe", "it might be", "possibly", "i'm not sure"]
    hedging_count = sum(1 for p in hedging if p in response.lower())
    checks["confidence"] = max(0.0, 1.0 - hedging_count * 0.2)

    # Check for empty or near-empty responses
    word_count = len(response.split())
    checks["substantive"] = 1.0 if word_count >= 10 else word_count / 10.0

    # Check for repetition (repeated sentences)
    sentences = [s.strip() for s in re.split(r'[.!?]+', response) if s.strip()]
    unique_ratio = len(set(sentences)) / max(len(sentences), 1)
    checks["non_repetitive"] = unique_ratio

    return checks

Drift Detection with Rolling Averages

Track quality scores over time and detect when they drift below baseline. A simple but effective approach compares a short-term rolling average against a long-term baseline.

from collections import deque
from datetime import datetime

class QualityDriftDetector:
    def __init__(
        self,
        baseline_window: int = 1000,   # Long-term baseline
        recent_window: int = 50,        # Short-term comparison
        alert_threshold: float = 0.05,  # Alert on 5% drop
    ):
        self.baseline_scores = deque(maxlen=baseline_window)
        self.recent_scores = deque(maxlen=recent_window)
        self.alert_threshold = alert_threshold
        self.alerts_sent = {}

    def record_score(self, dimension: str, score: float) -> dict | None:
        key = dimension
        if key not in self.alerts_sent:
            self.alerts_sent[key] = None

        self.baseline_scores.append(score)
        self.recent_scores.append(score)

        if len(self.baseline_scores) < 100 or len(self.recent_scores) < 20:
            return None  # Not enough data

        baseline_avg = sum(list(self.baseline_scores)[:len(self.baseline_scores) - len(self.recent_scores)]) / max(1, len(self.baseline_scores) - len(self.recent_scores))
        recent_avg = sum(self.recent_scores) / len(self.recent_scores)
        drift = baseline_avg - recent_avg

        if drift > self.alert_threshold:
            return {
                "dimension": dimension,
                "baseline_avg": round(baseline_avg, 3),
                "recent_avg": round(recent_avg, 3),
                "drift": round(drift, 3),
                "timestamp": datetime.utcnow().isoformat(),
            }
        return None

# Usage in the scoring pipeline
detector = QualityDriftDetector()

async def monitor_response(question: str, response: str, conversation_id: str):
    scores = await score_response(question, response, conversation_id)
    for score in scores:
        alert = detector.record_score(score.dimension.value, score.score)
        if alert:
            await send_quality_alert(alert)

Sampling Strategy

You do not need to score every response. A well-designed sampling strategy provides statistical coverage while controlling judge LLM costs.

import random
import hashlib

def should_sample(conversation_id: str, sample_rate: float = 0.05) -> bool:
    """Deterministic sampling based on conversation ID.
    The same conversation always gets the same decision, which
    enables reproducible analysis.
    """
    hash_value = int(hashlib.sha256(conversation_id.encode()).hexdigest(), 16)
    return (hash_value % 10000) / 10000.0 < sample_rate

FAQ

How do I detect quality degradation from a model provider update?

Run a fixed evaluation set — a curated list of 50-100 representative questions with known-good reference answers — against the production model on a daily schedule. Compare scores against the stored baseline. A sudden drop across the evaluation set strongly signals a model change, since your prompt and retrieval pipeline did not change.

Is using an LLM to judge another LLM reliable?

LLM-as-judge correlates well with human judgment on most quality dimensions when the judge model is at least as capable as the model being evaluated. The key is calibration: run your judge on a set of human-scored examples first and verify agreement. GPT-4o-mini as a judge of GPT-4o responses works well for relevance and completeness but can miss subtle factual errors that require domain expertise.

How much does a quality monitoring pipeline cost to run?

At a 5% sample rate with GPT-4o-mini as the judge, scoring adds roughly $0.50-$1.00 per 1000 production conversations. The heuristic checks are free. For most agent deployments, this cost is trivial compared to the cost of undetected quality degradation affecting user satisfaction and retention.

---

#QualityMonitoring #LLMEvaluation #DriftDetection #AIAgents #ProductionMonitoring #AgenticAI #LearnAI #AIEngineering