Auto-Scaling AI Agent Workers: Dynamic Scaling Based on Queue Depth and Latency

Why CPU-Based Auto-Scaling Fails for AI Agents

The default Kubernetes HorizontalPodAutoscaler scales based on CPU utilization. For AI agent workers, this metric is almost useless. Agent workers spend 80 to 95 percent of their time waiting on I/O — LLM API calls, database queries, tool executions. CPU sits at 5 to 15 percent even when the worker is at full capacity with all task slots occupied.

The result: your workers are overwhelmed and queue depth grows while HPA does nothing because CPU is below the threshold. You need to scale based on business-relevant metrics — queue depth, active sessions, and response latency.

KEDA: Event-Driven Auto-Scaling

KEDA (Kubernetes Event-Driven Autoscaling) extends Kubernetes with scalers that watch external event sources — message queues, databases, Prometheus metrics — and scale deployments accordingly. It supports scale-to-zero, which HPA does not:

flowchart LR
    REQ(["Request"])
    BATCH["Continuous batching<br/>vLLM scheduler"]
    PREF{"Prefill or<br/>decode?"}
    PRE["Prefill phase<br/>parallel attention"]
    DEC["Decode phase<br/>token by token"]
    KV[("Paged KV cache")]
    SAMP["Sampling<br/>top-p, temp"]
    STREAM["Stream tokens<br/>to client"]
    REQ --> BATCH --> PREF
    PREF -->|First token| PRE --> KV
    PREF -->|Next token| DEC
    KV --> DEC --> SAMP --> STREAM
    SAMP -->|EOS| DONE(["Response complete"])
    style BATCH fill:#4f46e5,stroke:#4338ca,color:#fff
    style KV fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style STREAM fill:#0ea5e9,stroke:#0369a1,color:#fff
    style DONE fill:#059669,stroke:#047857,color:#fff

# Install KEDA
# helm repo add kedacore https://kedacore.github.io/charts
# helm install keda kedacore/keda --namespace keda-system

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: agent-worker-scaler
  namespace: agents
spec:
  scaleTargetRef:
    name: agent-worker
  minReplicaCount: 2       # Always keep 2 warm
  maxReplicaCount: 100
  pollingInterval: 15       # Check every 15 seconds
  cooldownPeriod: 120       # Wait 2 min before scaling down
  advanced:
    restoreToOriginalReplicaCount: false
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
            - type: Pods
              value: 10
              periodSeconds: 30
        scaleDown:
          stabilizationWindowSeconds: 300
          policies:
            - type: Pods
              value: 3
              periodSeconds: 60
  triggers:
    # Scale based on RabbitMQ queue depth
    - type: rabbitmq
      metadata:
        host: amqp://user:pass@rabbitmq.agents:5672/
        queueName: agent_tasks
        mode: QueueLength
        value: "5"  # 5 messages per replica

This configuration means: if there are 50 messages in the queue, KEDA ensures 10 replicas are running (50 / 5 = 10). As the queue empties, it scales down. The stabilization window on scale-down prevents flapping during bursty workloads.

Multi-Metric Scaling

In practice, you want to scale based on multiple signals — queue depth for backlog pressure and latency for real-time pressure:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: agent-worker-multi-metric
  namespace: agents
spec:
  scaleTargetRef:
    name: agent-worker
  minReplicaCount: 2
  maxReplicaCount: 100
  triggers:
    # Trigger 1: Queue depth
    - type: rabbitmq
      metadata:
        host: amqp://user:pass@rabbitmq.agents:5672/
        queueName: agent_tasks
        mode: QueueLength
        value: "5"

    # Trigger 2: P95 response latency from Prometheus
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        metricName: agent_turn_latency_p95
        query: |
          histogram_quantile(0.95,
            sum(rate(agent_turn_latency_seconds_bucket[5m]))
            by (le)
          )
        threshold: "5"  # Scale when p95 > 5 seconds
        activationThreshold: "3"

    # Trigger 3: Active WebSocket connections
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        metricName: active_ws_connections
        query: |
          sum(active_agent_sessions)
        threshold: "200"  # 200 sessions per replica

KEDA evaluates all triggers and uses the one that requires the most replicas. If the queue is empty but latency is high, it still scales up. If latency is fine but the queue is deep, it also scales up.

Scale-to-Zero with Fast Warm-Up

For development or staging environments, or for specialized agent workers that handle rare task types, scale-to-zero saves significant cost. The challenge is cold-start latency — the first request after scale-up must wait for a pod to start:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: specialized-agent-scaler
spec:
  scaleTargetRef:
    name: specialized-agent-worker
  minReplicaCount: 0        # Scale to zero when idle
  maxReplicaCount: 20
  idleReplicaCount: 0       # Zero pods when no triggers active
  triggers:
    - type: rabbitmq
      metadata:
        host: amqp://user:pass@rabbitmq.agents:5672/
        queueName: specialized_tasks
        mode: QueueLength
        value: "1"
        activationValue: "1"  # Activate from zero at 1 message

Minimize cold-start time with these pod optimizations:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: specialized-agent-worker
spec:
  template:
    spec:
      containers:
        - name: worker
          image: agent-worker:latest
          # Preload models and connections at startup
          startupProbe:
            httpGet:
              path: /health/ready
              port: 8000
            initialDelaySeconds: 2
            periodSeconds: 2
            failureThreshold: 30
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"

In your application code, preload everything possible during startup:

from contextlib import asynccontextmanager
from fastapi import FastAPI

@asynccontextmanager
async def lifespan(app: FastAPI):
    """Preload resources at startup to minimize cold-start time."""
    # Initialize database connection pool
    await db.connect()

    # Pre-warm Redis connection
    await redis_client.ping()

    # Pre-load prompt templates into memory
    app.state.prompt_templates = await load_all_templates()

    # Pre-warm the HTTP client connection pool for LLM APIs
    async with httpx.AsyncClient() as client:
        await client.get("https://api.openai.com/v1/models")

    yield

    await db.disconnect()

app = FastAPI(lifespan=lifespan)

Monitoring Auto-Scaling Behavior

Track scaling events and their impact on performance:

# Prometheus metrics for scaling visibility
from prometheus_client import Gauge, Counter

scaling_events = Counter(
    "autoscaler_events_total",
    "Auto-scaling events",
    labelnames=["direction"],  # up or down
)

current_replicas = Gauge(
    "agent_worker_replicas",
    "Current number of agent worker replicas",
)

queue_depth = Gauge(
    "agent_task_queue_depth",
    "Current depth of the agent task queue",
)

Create a Grafana dashboard that correlates queue depth, replica count, and response latency over time. This visualization reveals whether your scaling is fast enough — if latency spikes before replicas increase, you need a more aggressive scale-up policy.

FAQ

How fast can KEDA scale up new pods?

KEDA checks triggers at the polling interval (default 30 seconds, configurable down to 1 second). After KEDA requests new replicas, Kubernetes takes 5 to 30 seconds to schedule and start a pod (depending on image pull policy and startup time). Total time from trigger to ready pod is typically 15 to 60 seconds.

Should I use KEDA or the built-in HPA with custom metrics?

KEDA is better for AI agent workloads because it supports scale-to-zero, has built-in scalers for message queues (no custom metrics adapter needed), and allows multiple trigger types on a single deployment. Use HPA only if you cannot install KEDA in your cluster.

What is the right queue depth threshold per replica?

Set it equal to the number of concurrent tasks each worker can handle. If each worker processes 5 tasks concurrently and you want no queuing delay, set the threshold to 5. If some queuing delay is acceptable, set it to 10 to 15 (one to two full batches waiting). Monitor the actual wait time and adjust.

---

#AutoScaling #KEDA #Kubernetes #AIAgents #HPA #QueueDepth #AgenticAI #LearnAI #AIEngineering