Agent Capacity Planning: Predicting Resource Needs for Growing Agent Workloads

Why Capacity Planning for AI Agents Is Different

AI agent workloads are fundamentally different from traditional web services. A single agent request might trigger 1 LLM call or 20, depending on reasoning complexity. Memory usage grows with conversation length. Tool calls create unpredictable downstream load. A 2x increase in user traffic can produce a 10x increase in LLM API calls.

Without proper capacity planning, you will either overpay for idle resources or face outages during traffic spikes.

Modeling Agent Resource Consumption

The first step is understanding what a single agent invocation actually consumes.

flowchart LR
    USERS(["Traffic"])
    LB["Geo LB plus<br/>Anycast"]
    EDGE["Edge cache plus<br/>rate limit"]
    APP["Stateless app pods<br/>HPA on QPS"]
    QUEUE[(Async work queue)]
    WORKER["Worker pool<br/>GPU or CPU"]
    CACHE[("Redis cache<br/>LLM responses")]
    DB[("Read replicas<br/>and primary")]
    OBS[(Observability)]
    USERS --> LB --> EDGE --> APP
    APP --> CACHE
    APP --> QUEUE --> WORKER
    APP --> DB
    APP --> OBS
    style LB fill:#4f46e5,stroke:#4338ca,color:#fff
    style WORKER fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style CACHE fill:#f59e0b,stroke:#d97706,color:#1f2937
    style OBS fill:#0ea5e9,stroke:#0369a1,color:#fff

from dataclasses import dataclass, field
from typing import List

@dataclass
class AgentResourceProfile:
    """Resource consumption for a single agent task execution."""
    avg_llm_calls: float
    avg_tool_calls: float
    avg_input_tokens: int
    avg_output_tokens: int
    avg_memory_mb: float
    avg_duration_seconds: float
    avg_db_queries: int
    p99_llm_calls: float
    p99_duration_seconds: float

@dataclass
class AgentCapacityModel:
    profiles: dict  # agent_type -> AgentResourceProfile

    def estimate_resources(self, requests_per_minute: dict) -> dict:
        total_llm_calls_per_min = 0
        total_memory_gb = 0
        total_db_queries_per_min = 0

        for agent_type, rpm in requests_per_minute.items():
            profile = self.profiles[agent_type]
            total_llm_calls_per_min += rpm * profile.avg_llm_calls
            concurrent = rpm * (profile.avg_duration_seconds / 60)
            total_memory_gb += concurrent * profile.avg_memory_mb / 1024
            total_db_queries_per_min += rpm * profile.avg_db_queries

        return {
            "llm_calls_per_minute": total_llm_calls_per_min,
            "concurrent_memory_gb": total_memory_gb,
            "db_queries_per_minute": total_db_queries_per_min,
            "llm_tokens_per_minute": self._estimate_tokens(requests_per_minute),
        }

    def _estimate_tokens(self, requests_per_minute: dict) -> int:
        total = 0
        for agent_type, rpm in requests_per_minute.items():
            p = self.profiles[agent_type]
            total += rpm * (p.avg_input_tokens + p.avg_output_tokens) * p.avg_llm_calls
        return total

# Example: build profiles from production metrics
model = AgentCapacityModel(profiles={
    "customer_support": AgentResourceProfile(
        avg_llm_calls=3.2, avg_tool_calls=1.8,
        avg_input_tokens=1200, avg_output_tokens=400,
        avg_memory_mb=128, avg_duration_seconds=8.5,
        avg_db_queries=4, p99_llm_calls=8, p99_duration_seconds=25,
    ),
    "data_analyst": AgentResourceProfile(
        avg_llm_calls=6.5, avg_tool_calls=4.2,
        avg_input_tokens=3000, avg_output_tokens=1500,
        avg_memory_mb=512, avg_duration_seconds=45,
        avg_db_queries=12, p99_llm_calls=15, p99_duration_seconds=120,
    ),
})

Notice the wide spread between average and p99 for the data analyst agent. This variance makes capacity planning harder than for traditional services.

Demand Forecasting

Use historical data to predict future agent workload. Combine time-series forecasting with business growth projections.

import numpy as np
from datetime import datetime, timedelta

class AgentDemandForecaster:
    def __init__(self, historical_rpm: list, growth_rate_monthly: float = 0.15):
        self.historical = np.array(historical_rpm)
        self.growth_rate = growth_rate_monthly

    def forecast_next_month(self) -> dict:
        # Baseline: current average with growth
        current_avg = np.mean(self.historical[-7:])  # last 7 days
        projected_avg = current_avg * (1 + self.growth_rate)

        # Peak: use historical peak ratio
        peak_ratio = np.max(self.historical) / np.mean(self.historical)
        projected_peak = projected_avg * peak_ratio

        # Burst: add safety margin for unexpected spikes
        burst_capacity = projected_peak * 1.5

        return {
            "avg_rpm": round(projected_avg, 1),
            "peak_rpm": round(projected_peak, 1),
            "burst_rpm": round(burst_capacity, 1),
            "growth_rate": self.growth_rate,
        }

    def months_until_limit(self, current_capacity_rpm: float) -> int:
        """Predict when you will hit capacity limits."""
        monthly_avg = np.mean(self.historical[-30:])
        months = 0
        projected = monthly_avg
        while projected < current_capacity_rpm and months < 36:
            months += 1
            projected *= (1 + self.growth_rate)
        return months

The months_until_limit method is your early warning system. If the answer is less than 3, start planning capacity expansion immediately.

Headroom and Scaling Triggers

Headroom is the gap between your current load and your maximum capacity. Scaling triggers define when to add resources.

# capacity-config.yaml
scaling:
  headroom_percentage: 30  # always maintain 30% spare capacity

  triggers:
    - name: "llm_concurrency_high"
      metric: "agent_concurrent_llm_calls"
      threshold: 80  # percent of rate limit
      action: "add_agent_pool_replicas"
      cooldown_seconds: 300

    - name: "memory_pressure"
      metric: "agent_pool_memory_utilization"
      threshold: 70  # percent
      action: "scale_up_node_pool"
      cooldown_seconds: 600

    - name: "queue_depth_growing"
      metric: "agent_task_queue_depth"
      threshold: 100  # pending tasks
      action: "add_agent_workers"
      cooldown_seconds: 120

    - name: "token_budget_approaching"
      metric: "daily_token_usage_percentage"
      threshold: 75
      action: "alert_team_and_throttle"
      cooldown_seconds: 3600

  cost_limits:
    max_daily_llm_spend: 500  # USD
    max_monthly_compute: 3000  # USD
    auto_scale_ceiling: 20  # max replicas

Token budget is a scaling constraint unique to AI systems. Unlike CPU or memory, LLM tokens have a direct dollar cost per unit. Your autoscaler must respect cost ceilings.

Building a Capacity Dashboard

class CapacityDashboard:
    def __init__(self, model: AgentCapacityModel, forecaster: AgentDemandForecaster):
        self.model = model
        self.forecaster = forecaster

    def generate_report(self, current_rpm: dict, limits: dict) -> dict:
        current_resources = self.model.estimate_resources(current_rpm)
        forecast = self.forecaster.forecast_next_month()

        peak_resources = self.model.estimate_resources(
            {k: v * (forecast["peak_rpm"] / forecast["avg_rpm"])
             for k, v in current_rpm.items()}
        )

        return {
            "current_utilization": {
                k: round(current_resources[k] / limits[k] * 100, 1)
                for k in limits
            },
            "projected_peak_utilization": {
                k: round(peak_resources[k] / limits[k] * 100, 1)
                for k in limits
            },
            "months_to_capacity": self.forecaster.months_until_limit(
                limits["llm_calls_per_minute"]
            ),
            "recommendation": self._recommend(peak_resources, limits),
        }

    def _recommend(self, peak: dict, limits: dict) -> str:
        max_util = max(peak[k] / limits[k] for k in limits)
        if max_util > 0.85:
            return "URGENT: Scale up immediately, peak will exceed capacity"
        elif max_util > 0.70:
            return "PLAN: Begin capacity expansion within 2 weeks"
        return "OK: Sufficient headroom for projected growth"

FAQ

How do I account for the unpredictable number of LLM calls per agent request?

Use percentile-based modeling instead of averages. Track the distribution of LLM calls per request and plan capacity for the p95 or p99 case, not the average. Your capacity model should include both average and peak profiles, and scaling decisions should use the peak profile.

What is a good headroom percentage for AI agent systems?

Aim for 30-40% headroom, higher than the typical 20% for traditional services. AI agents have higher variance in resource consumption, and LLM API latency can spike during provider-side load, causing requests to pile up. The extra headroom absorbs these bursts without degrading performance.

How do I plan capacity when LLM costs dominate compute costs?

Treat token budgets as a first-class capacity dimension alongside CPU and memory. Model cost per agent task, set daily and monthly spending limits, and build throttling mechanisms that activate when approaching budget limits. Negotiate committed-use discounts with LLM providers once your usage patterns stabilize.

---

#CapacityPlanning #AIAgents #Scaling #ResourceManagement #Infrastructure #AgenticAI #LearnAI #AIEngineering