--- title: "Evaluation Datasets for AI Agents: Building Ground Truth for Automated Testing" description: "Learn how to design, label, and maintain evaluation datasets for AI agents, covering dataset structure, diversity requirements, edge cases, and ongoing maintenance strategies." canonical: https://callsphere.ai/blog/evaluation-datasets-ai-agents-building-ground-truth category: "Learn Agentic AI" tags: ["Evaluation", "Datasets", "AI Agents", "Ground Truth", "Testing", "Python"] author: "CallSphere Team" published: 2026-03-17T00:00:00.000Z updated: 2026-05-06T01:02:42.452Z --- # Evaluation Datasets for AI Agents: Building Ground Truth for Automated Testing > Learn how to design, label, and maintain evaluation datasets for AI agents, covering dataset structure, diversity requirements, edge cases, and ongoing maintenance strategies. ## Why Evaluation Datasets Are the Foundation of Agent Quality An AI agent without an evaluation dataset is like a web service without tests — you only discover problems after users report them. Evaluation datasets provide ground truth: curated input-output pairs that define what correct behavior looks like. They enable automated regression testing, prompt comparison, and model migration decisions. The difference between a toy eval set and a production-grade one is coverage, labeling quality, and maintenance discipline. This guide walks through building eval datasets that actually catch real problems. ## Dataset Structure An eval dataset is a collection of test cases, each containing an input, the expected behavior, and metadata for slicing results. ```mermaid flowchart LR PR(["PR opened"]) UNIT["Unit tests"] EVAL["Eval harness PromptFoo or Braintrust"] GOLD[("Golden set 200 tagged cases")] JUDGE["LLM as judge plus regex graders"] SCORE["Aggregate score and per slice"] GATE{"Score regress more than 2 percent?"} BLOCK(["Block merge"]) MERGE(["Merge to main"]) PR --> UNIT --> EVAL --> GOLD --> JUDGE --> SCORE --> GATE GATE -->|Yes| BLOCK GATE -->|No| MERGE style EVAL fill:#4f46e5,stroke:#4338ca,color:#fff style GATE fill:#f59e0b,stroke:#d97706,color:#1f2937 style BLOCK fill:#dc2626,stroke:#b91c1c,color:#fff style MERGE fill:#059669,stroke:#047857,color:#fff ``` ```python from dataclasses import dataclass, field from enum import Enum from typing import Optional class Difficulty(str, Enum): EASY = "easy" MEDIUM = "medium" HARD = "hard" class Category(str, Enum): TOOL_USE = "tool_use" REASONING = "reasoning" REFUSAL = "refusal" MULTI_STEP = "multi_step" @dataclass class EvalCase: id: str input_text: str expected_output: str expected_tool_calls: list[str] = field(default_factory=list) category: Category = Category.REASONING difficulty: Difficulty = Difficulty.MEDIUM tags: list[str] = field(default_factory=list) notes: Optional[str] = None ``` Store eval cases in a structured format — JSON Lines works well because you can append new cases without rewriting the file. ```python import json from pathlib import Path def save_eval_dataset(cases: list[EvalCase], path: Path): with open(path, "w") as f: for case in cases: f.write(json.dumps(vars(case)) + "\n") def load_eval_dataset(path: Path) -> list[EvalCase]: cases = [] with open(path) as f: for line in f: data = json.loads(line) data["category"] = Category(data["category"]) data["difficulty"] = Difficulty(data["difficulty"]) cases.append(EvalCase(**data)) return cases ``` ## Designing for Diversity A common mistake is building eval sets that only test the happy path. Effective datasets cover five dimensions of diversity. ```python DIVERSITY_CHECKLIST = { "intent_types": [ "simple_question", # "What is X?" "multi_step_task", # "Find X, then do Y with it" "ambiguous_request", # "Help me with the thing" "out_of_scope", # "Write me a poem" (if agent is task-specific) "adversarial", # Prompt injection attempts ], "input_variations": [ "formal_english", "casual_with_typos", "non_english", "very_long_input", "empty_or_minimal", ], "expected_behaviors": [ "direct_answer", "tool_call", "clarifying_question", "polite_refusal", "multi_tool_chain", ], } def audit_coverage(cases: list[EvalCase]) -> dict: """Check which categories and difficulties are represented.""" coverage = { "categories": {}, "difficulties": {}, "total": len(cases), } for case in cases: coverage["categories"][case.category.value] = ( coverage["categories"].get(case.category.value, 0) + 1 ) coverage["difficulties"][case.difficulty.value] = ( coverage["difficulties"].get(case.difficulty.value, 0) + 1 ) return coverage ``` ## Labeling Best Practices Ground truth labels must be unambiguous. For open-ended outputs, use criteria-based labels instead of exact strings. ```python @dataclass class CriteriaLabel: """Define correctness as a checklist rather than an exact string.""" must_contain: list[str] = field(default_factory=list) must_not_contain: list[str] = field(default_factory=list) expected_tool: Optional[str] = None min_length: int = 0 max_length: int = 10_000 def evaluate(self, output: str, tool_calls: list[str]) -> dict: results = {} results["contains_required"] = all( kw.lower() in output.lower() for kw in self.must_contain ) results["avoids_forbidden"] = not any( kw.lower() in output.lower() for kw in self.must_not_contain ) results["correct_tool"] = ( self.expected_tool in tool_calls if self.expected_tool else True ) results["length_ok"] = self.min_length bool: last = datetime.fromisoformat(self.last_reviewed) return (datetime.now() - last).days > review_interval_days ``` Add new cases from production failures — every bug report is a potential eval case. Remove cases that no longer represent valid behavior. ## FAQ ### How many eval cases do I need? Start with 50-100 cases that cover your major use cases and known edge cases. Grow the dataset over time by adding cases from production failures. Quality and diversity matter more than raw count. ### Should I use synthetic data to generate eval cases? Synthetic generation with an LLM is useful for initial dataset bootstrapping, but always have a human review and correct the labels. LLM-generated ground truth inherits the model's biases and errors. ### How do I handle eval cases where multiple answers are correct? Use criteria-based labels (must contain certain keywords, must call certain tools) instead of exact string matching. This accommodates valid variation in phrasing while still catching incorrect behavior. --- #Evaluation #Datasets #AIAgents #GroundTruth #Testing #Python #AgenticAI #LearnAI #AIEngineering --- Source: https://callsphere.ai/blog/evaluation-datasets-ai-agents-building-ground-truth