Benchmarking and Profiling AI Agent Performance: Tools, Methodology, and Baseline Setting

Why You Need Agent Benchmarks

Without benchmarks, you cannot answer basic questions about your agent: Is it getting faster or slower? Did the last deployment improve response quality? How does it perform under load? Performance optimization without measurement is guesswork.

Agent benchmarks differ from traditional API benchmarks because they must measure both computational performance (latency, throughput, memory) and behavioral performance (response quality, tool usage accuracy, task completion rate). You need both to have a complete picture.

Defining Baseline Metrics

Start by defining the metrics that matter for your specific agent and establishing baseline values.

flowchart LR
    PR(["PR opened"])
    UNIT["Unit tests"]
    EVAL["Eval harness<br/>PromptFoo or Braintrust"]
    GOLD[("Golden set<br/>200 tagged cases")]
    JUDGE["LLM as judge<br/>plus regex graders"]
    SCORE["Aggregate score<br/>and per slice"]
    GATE{"Score regress<br/>more than 2 percent?"}
    BLOCK(["Block merge"])
    MERGE(["Merge to main"])
    PR --> UNIT --> EVAL --> GOLD --> JUDGE --> SCORE --> GATE
    GATE -->|Yes| BLOCK
    GATE -->|No| MERGE
    style EVAL fill:#4f46e5,stroke:#4338ca,color:#fff
    style GATE fill:#f59e0b,stroke:#d97706,color:#1f2937
    style BLOCK fill:#dc2626,stroke:#b91c1c,color:#fff
    style MERGE fill:#059669,stroke:#047857,color:#fff

from dataclasses import dataclass, field
from typing import Optional
import time
import statistics

@dataclass
class AgentMetrics:
    """Metrics for a single agent run."""
    # Latency
    time_to_first_token_ms: float = 0
    total_response_time_ms: float = 0

    # Resource usage
    llm_calls: int = 0
    tool_calls: int = 0
    total_input_tokens: int = 0
    total_output_tokens: int = 0

    # Quality
    task_completed: bool = False
    tool_accuracy: float = 0.0  # % of tool calls that were correct

    # Cost
    estimated_cost_usd: float = 0.0

@dataclass
class BenchmarkBaseline:
    """Baseline performance expectations."""
    max_ttft_ms: float = 1000
    max_total_time_ms: float = 10000
    min_task_completion_rate: float = 0.90
    max_avg_llm_calls: float = 5
    max_cost_per_query_usd: float = 0.05

    def check(self, metrics: AgentMetrics) -> dict[str, bool]:
        return {
            "ttft_ok": metrics.time_to_first_token_ms <= self.max_ttft_ms,
            "total_time_ok": metrics.total_response_time_ms <= self.max_total_time_ms,
            "llm_calls_ok": metrics.llm_calls <= self.max_avg_llm_calls,
            "cost_ok": metrics.estimated_cost_usd <= self.max_cost_per_query_usd,
        }

Building a Benchmark Test Suite

A good benchmark suite covers representative queries across your agent's capabilities. Include easy, medium, and hard cases.

from dataclasses import dataclass
from enum import Enum

class Difficulty(Enum):
    EASY = "easy"        # Single tool call, direct answer
    MEDIUM = "medium"    # 2-3 tool calls, some reasoning
    HARD = "hard"        # 4+ tool calls, multi-step reasoning

@dataclass
class BenchmarkCase:
    name: str
    query: str
    difficulty: Difficulty
    expected_tools: list[str]
    expected_answer_contains: list[str]
    max_time_ms: float

BENCHMARK_SUITE = [
    BenchmarkCase(
        name="simple_lookup",
        query="What is the return policy?",
        difficulty=Difficulty.EASY,
        expected_tools=["search_knowledge_base"],
        expected_answer_contains=["30 days", "refund"],
        max_time_ms=3000,
    ),
    BenchmarkCase(
        name="order_status",
        query="What is the status of order #12345?",
        difficulty=Difficulty.MEDIUM,
        expected_tools=["lookup_order", "get_shipping_status"],
        expected_answer_contains=["shipped", "tracking"],
        max_time_ms=6000,
    ),
    BenchmarkCase(
        name="complex_resolution",
        query="I received a damaged item from order #12345, I want a replacement shipped to my new address at 123 Main St.",
        difficulty=Difficulty.HARD,
        expected_tools=[
            "lookup_order", "create_return", "update_address", "create_replacement"
        ],
        expected_answer_contains=["replacement", "return label"],
        max_time_ms=15000,
    ),
]

Running Benchmarks with Instrumented Agent

Wrap your agent with instrumentation to capture metrics during each benchmark run.

import asyncio
import time
from typing import Any

class InstrumentedAgentRunner:
    def __init__(self, agent, tool_registry: dict):
        self.agent = agent
        self.tools = tool_registry

    async def run_benchmark(self, suite: list[BenchmarkCase]) -> list[dict]:
        results = []
        for case in suite:
            metrics = await self._run_single(case)
            baseline = BenchmarkBaseline()
            checks = baseline.check(metrics)

            results.append({
                "case": case.name,
                "difficulty": case.difficulty.value,
                "metrics": metrics,
                "passed_baseline": all(checks.values()),
                "checks": checks,
            })
        return results

    async def _run_single(self, case: BenchmarkCase) -> AgentMetrics:
        metrics = AgentMetrics()

        t_start = time.perf_counter()
        # Run the agent with the benchmark query
        result = await self.agent.run(
            case.query,
            on_tool_call=lambda name, args: self._track_tool(metrics, name),
            on_first_token=lambda: self._track_ttft(metrics, t_start),
        )
        t_end = time.perf_counter()

        metrics.total_response_time_ms = (t_end - t_start) * 1000

        # Check task completion
        answer = result.lower()
        metrics.task_completed = all(
            keyword.lower() in answer for keyword in case.expected_answer_contains
        )

        # Check tool accuracy
        actual_tools = metrics._tool_names if hasattr(metrics, "_tool_names") else []
        correct = sum(1 for t in actual_tools if t in case.expected_tools)
        metrics.tool_accuracy = correct / max(len(actual_tools), 1)

        return metrics

    def _track_tool(self, metrics: AgentMetrics, tool_name: str):
        metrics.tool_calls += 1
        if not hasattr(metrics, "_tool_names"):
            metrics._tool_names = []
        metrics._tool_names.append(tool_name)

    def _track_ttft(self, metrics: AgentMetrics, start_time: float):
        metrics.time_to_first_token_ms = (time.perf_counter() - start_time) * 1000

Profiling with cProfile and Line Profiler

For deep performance analysis, use Python's profiling tools to find exactly where time is spent.

import cProfile
import pstats
import io
from functools import wraps

def profile_async(func):
    """Decorator to profile an async function."""
    @wraps(func)
    async def wrapper(*args, **kwargs):
        profiler = cProfile.Profile()
        profiler.enable()

        result = await func(*args, **kwargs)

        profiler.disable()

        # Print top 20 functions by cumulative time
        stream = io.StringIO()
        stats = pstats.Stats(profiler, stream=stream)
        stats.sort_stats("cumulative")
        stats.print_stats(20)
        print(stream.getvalue())

        return result
    return wrapper

# Usage
@profile_async
async def profiled_agent_run(agent, query: str):
    return await agent.run(query)

For more granular analysis, use py-spy to profile running processes without modifying code:

# Install: pip install py-spy
# Profile a running agent server:
# py-spy record -o profile.svg --pid <PID> --duration 30

# Or profile a specific script:
# py-spy record -o profile.svg -- python run_benchmark.py

# The output is a flamegraph SVG showing where time is spent

Regression Tracking: Catching Performance Degradation

Store benchmark results over time and compare against historical baselines to catch regressions.

import json
import datetime
from pathlib import Path

class RegressionTracker:
    def __init__(self, results_dir: str = "./benchmark_results"):
        self.results_dir = Path(results_dir)
        self.results_dir.mkdir(exist_ok=True)

    def save_run(self, results: list[dict], git_sha: str):
        timestamp = datetime.datetime.now().isoformat()
        filename = f"bench_{timestamp}_{git_sha[:8]}.json"

        data = {
            "timestamp": timestamp,
            "git_sha": git_sha,
            "results": results,
            "summary": self._summarize(results),
        }

        filepath = self.results_dir / filename
        filepath.write_text(json.dumps(data, indent=2, default=str))
        return filepath

    def _summarize(self, results: list[dict]) -> dict:
        times = [r["metrics"].total_response_time_ms for r in results]
        return {
            "total_cases": len(results),
            "passed": sum(1 for r in results if r["passed_baseline"]),
            "avg_response_time_ms": sum(times) / len(times) if times else 0,
            "p95_response_time_ms": sorted(times)[int(len(times) * 0.95)] if times else 0,
        }

    def check_regression(self, current: dict, threshold_pct: float = 15.0) -> list[str]:
        """Compare current run against the last known good run."""
        previous_files = sorted(self.results_dir.glob("bench_*.json"))
        if not previous_files:
            return []

        previous = json.loads(previous_files[-1].read_text())
        warnings = []

        prev_avg = previous["summary"]["avg_response_time_ms"]
        curr_avg = current["summary"]["avg_response_time_ms"]

        if prev_avg > 0:
            pct_change = ((curr_avg - prev_avg) / prev_avg) * 100
            if pct_change > threshold_pct:
                warnings.append(
                    f"Average response time regressed by {pct_change:.1f}% "
                    f"({prev_avg:.0f}ms -> {curr_avg:.0f}ms)"
                )

        prev_pass_rate = previous["summary"]["passed"] / max(previous["summary"]["total_cases"], 1)
        curr_pass_rate = current["summary"]["passed"] / max(current["summary"]["total_cases"], 1)

        if curr_pass_rate < prev_pass_rate - 0.05:
            warnings.append(
                f"Pass rate dropped from {prev_pass_rate:.0%} to {curr_pass_rate:.0%}"
            )

        return warnings

Load Testing Your Agent

Benchmark single-query performance first, then test under concurrent load to find the breaking point.

import asyncio
import time

async def load_test(agent, queries: list[str], concurrency: int = 10) -> dict:
    """Run queries at the specified concurrency level."""
    semaphore = asyncio.Semaphore(concurrency)
    results = []

    async def run_one(query: str):
        async with semaphore:
            t_start = time.perf_counter()
            try:
                response = await agent.run(query)
                duration = (time.perf_counter() - t_start) * 1000
                results.append({"status": "ok", "duration_ms": duration})
            except Exception as e:
                duration = (time.perf_counter() - t_start) * 1000
                results.append({"status": "error", "duration_ms": duration, "error": str(e)})

    tasks = [run_one(q) for q in queries]
    await asyncio.gather(*tasks)

    durations = [r["duration_ms"] for r in results if r["status"] == "ok"]
    errors = [r for r in results if r["status"] == "error"]

    return {
        "total_requests": len(results),
        "successful": len(durations),
        "failed": len(errors),
        "avg_ms": sum(durations) / len(durations) if durations else 0,
        "p50_ms": sorted(durations)[len(durations) // 2] if durations else 0,
        "p95_ms": sorted(durations)[int(len(durations) * 0.95)] if durations else 0,
        "p99_ms": sorted(durations)[int(len(durations) * 0.99)] if durations else 0,
        "error_rate": len(errors) / len(results) if results else 0,
    }

# Run increasing concurrency to find the breaking point
for concurrency in [1, 5, 10, 25, 50]:
    result = await load_test(agent, queries * 10, concurrency=concurrency)
    print(f"Concurrency {concurrency}: avg={result['avg_ms']:.0f}ms, "
          f"p95={result['p95_ms']:.0f}ms, errors={result['error_rate']:.1%}")

FAQ

How often should I run performance benchmarks?

Run the full benchmark suite in your CI/CD pipeline on every pull request that touches agent code, tool implementations, or prompt templates. Run the load test suite weekly or before major releases. Store all results for trend analysis.

What is a good P95 latency target for an AI agent?

For conversational agents, a P95 of 5 seconds end-to-end (including LLM inference) is a reasonable starting target. This means 95% of queries complete within 5 seconds. For simple lookup queries, aim for P95 under 3 seconds. For complex multi-step tasks, P95 under 15 seconds is acceptable if the agent streams intermediate progress to the user.

How do I benchmark quality alongside performance?

Include expected-output assertions in your benchmark cases. After each run, check whether the response contains required keywords, uses the correct tools, and avoids known failure patterns. Track quality metrics (task completion rate, tool accuracy) on the same dashboard as latency metrics so you can catch quality-speed tradeoffs immediately.

---

#Benchmarking #Profiling #Metrics #Testing #Python #AgenticAI #LearnAI #AIEngineering