Infrastructure Cost Optimization for AI Agents: Right-Sizing Compute and Storage

Infrastructure Costs Are the Silent Budget Killer

Teams obsess over LLM token costs while running oversized compute instances 24/7. For many AI agent deployments, infrastructure costs (compute, storage, networking) rival or exceed LLM API costs. A single m5.2xlarge instance running idle at night costs $277/month. Multiply that by a few services, add a vector database cluster, and infrastructure alone can hit $2,000–$5,000/month before you send a single API call.

The fix is systematic: measure actual resource usage, right-size instances, implement auto-scaling, and use pricing tiers (spot, reserved) strategically.

Measuring Resource Utilization

Before optimizing, you need to know what you are actually using.

flowchart LR
    GIT(["Git push"])
    CI["GitHub Actions<br/>build plus test"]
    REG[("Container registry<br/>GHCR or ECR")]
    HELM["Helm chart<br/>values per env"]
    K8S{"Kubernetes cluster"}
    DEP["Deployment<br/>rolling update"]
    SVC["Service plus Ingress"]
    HPA["HPA<br/>CPU and queue depth"]
    POD[("Inference pods<br/>GPU node pool")]
    USERS(["Production traffic"])
    GIT --> CI --> REG --> HELM --> K8S
    K8S --> DEP --> POD
    K8S --> SVC --> POD
    K8S --> HPA --> POD
    SVC --> USERS
    style CI fill:#4f46e5,stroke:#4338ca,color:#fff
    style POD fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style USERS fill:#059669,stroke:#047857,color:#fff

import psutil
import time
from dataclasses import dataclass, field
from typing import List

@dataclass
class ResourceSnapshot:
    timestamp: float
    cpu_percent: float
    memory_percent: float
    memory_used_mb: float
    disk_used_percent: float
    network_bytes_sent: int
    network_bytes_recv: int

class ResourceMonitor:
    def __init__(self):
        self.snapshots: List[ResourceSnapshot] = []

    def capture(self) -> ResourceSnapshot:
        net = psutil.net_io_counters()
        snapshot = ResourceSnapshot(
            timestamp=time.time(),
            cpu_percent=psutil.cpu_percent(interval=1),
            memory_percent=psutil.virtual_memory().percent,
            memory_used_mb=psutil.virtual_memory().used / (1024 * 1024),
            disk_used_percent=psutil.disk_usage("/").percent,
            network_bytes_sent=net.bytes_sent,
            network_bytes_recv=net.bytes_recv,
        )
        self.snapshots.append(snapshot)
        return snapshot

    def utilization_summary(self) -> dict:
        if not self.snapshots:
            return {}
        return {
            "avg_cpu": round(sum(s.cpu_percent for s in self.snapshots) / len(self.snapshots), 1),
            "max_cpu": round(max(s.cpu_percent for s in self.snapshots), 1),
            "avg_memory": round(
                sum(s.memory_percent for s in self.snapshots) / len(self.snapshots), 1
            ),
            "max_memory": round(max(s.memory_percent for s in self.snapshots), 1),
            "p95_cpu": round(sorted(s.cpu_percent for s in self.snapshots)[
                int(len(self.snapshots) * 0.95)
            ], 1),
            "samples": len(self.snapshots),
        }

    def is_oversized(self) -> dict:
        summary = self.utilization_summary()
        return {
            "cpu_oversized": summary.get("p95_cpu", 0) < 30,
            "memory_oversized": summary.get("max_memory", 0) < 40,
            "recommendation": self._recommend(summary),
        }

    def _recommend(self, summary: dict) -> str:
        if summary.get("p95_cpu", 0) < 20 and summary.get("max_memory", 0) < 30:
            return "Strongly consider downsizing to a smaller instance"
        elif summary.get("p95_cpu", 0) < 40:
            return "Moderate opportunity to downsize"
        return "Current sizing appears appropriate"

Auto-Scaling Configuration

AI agent traffic follows predictable patterns: high during business hours, low at night. Auto-scaling matches capacity to demand.

from dataclasses import dataclass

@dataclass
class ScalingPolicy:
    min_replicas: int
    max_replicas: int
    target_cpu_percent: int
    target_memory_percent: int
    scale_up_cooldown_seconds: int = 60
    scale_down_cooldown_seconds: int = 300

ENVIRONMENT_POLICIES = {
    "production": ScalingPolicy(
        min_replicas=2,
        max_replicas=20,
        target_cpu_percent=60,
        target_memory_percent=70,
        scale_up_cooldown_seconds=30,
        scale_down_cooldown_seconds=300,
    ),
    "staging": ScalingPolicy(
        min_replicas=1,
        max_replicas=3,
        target_cpu_percent=70,
        target_memory_percent=80,
    ),
}

def generate_k8s_hpa(name: str, policy: ScalingPolicy) -> dict:
    return {
        "apiVersion": "autoscaling/v2",
        "kind": "HorizontalPodAutoscaler",
        "metadata": {"name": f"{name}-hpa"},
        "spec": {
            "scaleTargetRef": {
                "apiVersion": "apps/v1",
                "kind": "Deployment",
                "name": name,
            },
            "minReplicas": policy.min_replicas,
            "maxReplicas": policy.max_replicas,
            "metrics": [
                {
                    "type": "Resource",
                    "resource": {
                        "name": "cpu",
                        "target": {
                            "type": "Utilization",
                            "averageUtilization": policy.target_cpu_percent,
                        },
                    },
                },
            ],
            "behavior": {
                "scaleDown": {
                    "stabilizationWindowSeconds": policy.scale_down_cooldown_seconds,
                },
            },
        },
    }

Spot Instance Strategy

Spot instances offer 60–90% savings over on-demand pricing but can be interrupted. Use them for stateless, fault-tolerant agent workloads.

@dataclass
class SpotStrategy:
    on_demand_base: int  # minimum on-demand instances for reliability
    spot_ratio: float    # percentage of additional capacity to run on spot
    instance_types: List[str]  # diversify across types for availability
    fallback_to_on_demand: bool = True

RECOMMENDED_STRATEGIES = {
    "agent_workers": SpotStrategy(
        on_demand_base=2,
        spot_ratio=0.70,
        instance_types=["m5.large", "m5a.large", "m6i.large"],
    ),
    "batch_processors": SpotStrategy(
        on_demand_base=0,
        spot_ratio=1.0,
        instance_types=["c5.xlarge", "c5a.xlarge", "c6i.xlarge"],
    ),
    "vector_database": SpotStrategy(
        on_demand_base=3,
        spot_ratio=0.0,  # never use spot for stateful data stores
        instance_types=["r5.xlarge"],
    ),
}

Storage Optimization

AI agent systems generate large volumes of logs, traces, and conversation histories. Implement tiered storage with automatic lifecycle policies.

STORAGE_TIERS = {
    "hot": {
        "retention_days": 7,
        "storage_type": "SSD",
        "cost_per_gb_month": 0.10,
        "use_for": ["active conversations", "recent traces", "cache"],
    },
    "warm": {
        "retention_days": 90,
        "storage_type": "HDD / S3 Standard",
        "cost_per_gb_month": 0.023,
        "use_for": ["historical conversations", "analytics data"],
    },
    "cold": {
        "retention_days": 365,
        "storage_type": "S3 Glacier",
        "cost_per_gb_month": 0.004,
        "use_for": ["audit logs", "compliance archives"],
    },
}

FAQ

How do I decide between right-sizing and auto-scaling?

Do both. Right-size first to establish the correct baseline instance type, then add auto-scaling to handle demand fluctuations. Right-sizing without auto-scaling wastes money during off-peak hours. Auto-scaling on oversized instances scales the wrong resource — you end up adding more capacity than needed per replica.

Are spot instances safe for production AI agent workloads?

Yes, for stateless worker processes that can tolerate restarts. Run a base layer of on-demand instances (enough to handle minimum expected traffic) and use spot for burst capacity. Never use spot for stateful services like databases, vector stores, or in-memory caches that would lose data on termination.

How much can I realistically save with infrastructure optimization?

Teams that have never optimized typically find 30–50% savings from right-sizing alone. Adding auto-scaling saves another 15–25% on variable workloads. Spot instances for eligible workloads add another 20–30% savings on those specific instances. Combined, total infrastructure cost reductions of 40–60% are common.

---

#Infrastructure #CostOptimization #AutoScaling #CloudComputing #Kubernetes #AgenticAI #LearnAI #AIEngineering