Regression Testing for Prompt Changes: Catching Quality Drops Before Deployment

The Hidden Risk of Prompt Changes

Changing a single word in a system prompt can cause cascading quality regressions. A developer tweaks the prompt to fix one edge case and unknowingly breaks ten others. Without regression testing, these regressions only surface when users complain.

Prompt regression testing means running your evaluation dataset against both the old and new prompt, comparing scores, and blocking deployment when quality drops below a threshold. This is the prompt engineering equivalent of running your test suite before merging a pull request.

Prompt Versioning

Track prompts as versioned artifacts so you can compare any two versions.

flowchart TD
    SPEC(["Task spec"])
    SYSTEM["System prompt<br/>role plus rules"]
    SHOTS["Few shot examples<br/>3 to 5"]
    VARS["Variable injection<br/>Jinja or f-string"]
    COT["Chain of thought<br/>or scratchpad"]
    CONSTR["Output constraint<br/>JSON schema"]
    LLM["LLM call"]
    EVAL["Offline eval<br/>LLM as judge plus regex"]
    GATE{"Score over<br/>threshold?"}
    COMMIT(["Promote to prod<br/>version pinned"])
    REVISE(["Revise prompt"])
    SPEC --> SYSTEM --> SHOTS --> VARS --> COT --> CONSTR --> LLM --> EVAL --> GATE
    GATE -->|Yes| COMMIT
    GATE -->|No| REVISE --> SYSTEM
    style LLM fill:#4f46e5,stroke:#4338ca,color:#fff
    style EVAL fill:#f59e0b,stroke:#d97706,color:#1f2937
    style COMMIT fill:#059669,stroke:#047857,color:#fff

import hashlib
import json
from dataclasses import dataclass, field
from datetime import datetime
from pathlib import Path

@dataclass
class PromptVersion:
    name: str
    version: int
    content: str
    created_at: str = field(default_factory=lambda: datetime.now().isoformat())
    metadata: dict = field(default_factory=dict)

    @property
    def fingerprint(self) -> str:
        return hashlib.sha256(self.content.encode()).hexdigest()[:12]

class PromptRegistry:
    def __init__(self, storage_dir: Path):
        self.storage_dir = storage_dir
        self.storage_dir.mkdir(parents=True, exist_ok=True)

    def save(self, prompt: PromptVersion):
        path = self.storage_dir / f"{prompt.name}_v{prompt.version}.json"
        path.write_text(json.dumps(vars(prompt), indent=2))

    def load(self, name: str, version: int) -> PromptVersion:
        path = self.storage_dir / f"{name}_v{version}.json"
        data = json.loads(path.read_text())
        return PromptVersion(**data)

    def latest_version(self, name: str) -> int:
        versions = [
            int(p.stem.split("_v")[1])
            for p in self.storage_dir.glob(f"{name}_v*.json")
        ]
        return max(versions) if versions else 0

Building a Regression Test Suite

A regression suite runs the same eval cases against two prompt versions and compares results.

from dataclasses import dataclass

@dataclass
class RegressionResult:
    case_id: str
    input_text: str
    baseline_score: int
    candidate_score: int
    delta: int
    baseline_output: str
    candidate_output: str

def run_regression_suite(
    eval_cases: list[dict],
    baseline_prompt: str,
    candidate_prompt: str,
    agent_fn,
    judge_fn,
) -> list[RegressionResult]:
    results = []
    for case in eval_cases:
        baseline_output = agent_fn(case["input"], system_prompt=baseline_prompt)
        candidate_output = agent_fn(case["input"], system_prompt=candidate_prompt)

        baseline_score = judge_fn(case["input"], baseline_output, case["expected"])
        candidate_score = judge_fn(case["input"], candidate_output, case["expected"])

        results.append(RegressionResult(
            case_id=case["id"],
            input_text=case["input"],
            baseline_score=baseline_score,
            candidate_score=candidate_score,
            delta=candidate_score - baseline_score,
            baseline_output=baseline_output,
            candidate_output=candidate_output,
        ))
    return results

Diff Reporting

Generate human-readable reports that highlight regressions and improvements.

def generate_regression_report(results: list[RegressionResult]) -> str:
    regressions = [r for r in results if r.delta < 0]
    improvements = [r for r in results if r.delta > 0]
    unchanged = [r for r in results if r.delta == 0]

    avg_baseline = sum(r.baseline_score for r in results) / len(results)
    avg_candidate = sum(r.candidate_score for r in results) / len(results)

    lines = [
        "# Prompt Regression Report",
        f"Total cases: {len(results)}",
        f"Baseline avg score: {avg_baseline:.2f}",
        f"Candidate avg score: {avg_candidate:.2f}",
        f"Delta: {avg_candidate - avg_baseline:+.2f}",
        "",
        f"Regressions: {len(regressions)}",
        f"Improvements: {len(improvements)}",
        f"Unchanged: {len(unchanged)}",
    ]

    if regressions:
        lines.append("\n## Regressions (score decreased)")
        for r in sorted(regressions, key=lambda x: x.delta):
            lines.append(f"  [{r.case_id}] {r.baseline_score} -> {r.candidate_score} "
                         f"({r.delta:+d}): {r.input_text[:80]}")

    if improvements:
        lines.append("\n## Improvements (score increased)")
        for r in sorted(improvements, key=lambda x: x.delta, reverse=True):
            lines.append(f"  [{r.case_id}] {r.baseline_score} -> {r.candidate_score} "
                         f"({r.delta:+d}): {r.input_text[:80]}")

    return "\n".join(lines)

CI Integration

Block merges when a prompt change causes quality regression beyond a threshold.

import sys

def check_regression_gate(
    results: list[RegressionResult],
    max_regression_count: int = 2,
    min_avg_score: float = 3.5,
) -> bool:
    regressions = [r for r in results if r.delta < -1]  # significant drops only
    avg_candidate = sum(r.candidate_score for r in results) / len(results)

    if len(regressions) > max_regression_count:
        print(f"FAIL: {len(regressions)} significant regressions "
              f"(max allowed: {max_regression_count})")
        return False
    if avg_candidate < min_avg_score:
        print(f"FAIL: Average score {avg_candidate:.2f} "
              f"below threshold {min_avg_score}")
        return False

    print(f"PASS: {len(regressions)} regressions, avg score {avg_candidate:.2f}")
    return True

# In CI script:
# if not check_regression_gate(results):
#     sys.exit(1)

A GitHub Actions workflow for prompt regression:

# .github/workflows/prompt-regression.yml
on:
  pull_request:
    paths:
      - "prompts/**"

jobs:
  regression-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 2
      - run: pip install -e ".[test]"
      - run: python -m pytest tests/regression/ -v --tb=long
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY_TEST }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: regression-report
          path: reports/regression_*.txt

FAQ

How many eval cases do I need for regression testing?

A minimum of 30-50 cases gives you statistical confidence. Aim for at least 5 cases per major use case your agent handles. The suite should run in under 10 minutes to keep the CI feedback loop fast.

What threshold should I use for blocking deployments?

Start conservative: block on any regression of 2 or more points on a 5-point scale, or if more than 10% of cases regress at all. Relax the threshold as you gain confidence in your eval dataset quality.

Can I regression test without an LLM-as-Judge?

Yes. For structured outputs (JSON, tool calls), use deterministic assertions. For text outputs, use keyword matching or embedding similarity. LLM-as-Judge adds cost but gives higher-quality evaluation for open-ended responses.

---

#RegressionTesting #PromptEngineering #AIAgents #CICD #Python #QualityAssurance #AgenticAI #LearnAI #AIEngineering