Kubernetes Persistent Volumes for AI Agent State: PVC Patterns and Storage Classes

Why AI Agents Need Persistent Storage

AI agents often maintain state that must survive Pod restarts. Local vector databases like ChromaDB or FAISS store embeddings on disk. Conversation history logs feed into analytics pipelines. Model weight caches prevent expensive re-downloads. Without persistent storage, all of this vanishes when Kubernetes reschedules a Pod to a different node.

Persistent Volume Claims (PVCs)

A PersistentVolumeClaim requests storage from the cluster. You specify the size and access mode, and Kubernetes provisions the volume automatically through a StorageClass.

flowchart LR
    GIT(["Git push"])
    CI["GitHub Actions<br/>build plus test"]
    REG[("Container registry<br/>GHCR or ECR")]
    HELM["Helm chart<br/>values per env"]
    K8S{"Kubernetes cluster"}
    DEP["Deployment<br/>rolling update"]
    SVC["Service plus Ingress"]
    HPA["HPA<br/>CPU and queue depth"]
    POD[("Inference pods<br/>GPU node pool")]
    USERS(["Production traffic"])
    GIT --> CI --> REG --> HELM --> K8S
    K8S --> DEP --> POD
    K8S --> SVC --> POD
    K8S --> HPA --> POD
    SVC --> USERS
    style CI fill:#4f46e5,stroke:#4338ca,color:#fff
    style POD fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style USERS fill:#059669,stroke:#047857,color:#fff

# vector-store-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vector-store
  namespace: ai-agents
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 50Gi

Mount the PVC in your Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-agent-with-vectordb
  namespace: ai-agents
spec:
  replicas: 1  # ReadWriteOnce limits to one Pod
  selector:
    matchLabels:
      app: ai-agent-vectordb
  template:
    metadata:
      labels:
        app: ai-agent-vectordb
    spec:
      containers:
        - name: agent
          image: myregistry/ai-agent:1.0.0
          volumeMounts:
            - name: vector-data
              mountPath: /data/vectordb
            - name: model-cache
              mountPath: /data/models
      volumes:
        - name: vector-data
          persistentVolumeClaim:
            claimName: vector-store
        - name: model-cache
          persistentVolumeClaim:
            claimName: model-cache

Storage Classes

StorageClasses define the type and performance tier of storage. Most cloud providers offer multiple classes:

# fast-ssd-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  iopsPerGB: "50"
  throughput: "250"
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

Key parameters for AI workloads: type: gp3 provides consistent SSD performance. reclaimPolicy: Retain keeps the volume when the PVC is deleted — critical for valuable embedding data. allowVolumeExpansion: true lets you grow the volume without recreating it. WaitForFirstConsumer binds the volume to the same availability zone as the Pod.

StatefulSets for Per-Replica Storage

When each agent replica needs its own dedicated storage, use a StatefulSet with volumeClaimTemplates:

# agent-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: agent-workers
  namespace: ai-agents
spec:
  serviceName: agent-workers
  replicas: 3
  selector:
    matchLabels:
      app: agent-worker
  template:
    metadata:
      labels:
        app: agent-worker
    spec:
      containers:
        - name: agent
          image: myregistry/ai-agent:1.0.0
          volumeMounts:
            - name: agent-data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: agent-data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-ssd
        resources:
          requests:
            storage: 20Gi

This creates three Pods (agent-workers-0, agent-workers-1, agent-workers-2) each with their own 20Gi PVC. The PVCs persist across Pod rescheduling and scale-down events.

Python Agent Using Persistent Storage

import os
from pathlib import Path
import chromadb

DATA_DIR = Path(os.environ.get("DATA_DIR", "/data/vectordb"))

def get_vector_store():
    """Initialize ChromaDB with persistent storage."""
    client = chromadb.PersistentClient(path=str(DATA_DIR))
    collection = client.get_or_create_collection(
        name="agent_knowledge",
        metadata={"hnsw:space": "cosine"}
    )
    return collection

def cache_model_weights(model_name: str, weights_path: Path):
    """Cache downloaded model weights to persistent volume."""
    cache_dir = Path("/data/models") / model_name
    if cache_dir.exists():
        print(f"Model {model_name} already cached")
        return cache_dir
    cache_dir.mkdir(parents=True, exist_ok=True)
    # Download and save to persistent storage
    return cache_dir

Backup Strategies

Use VolumeSnapshots to back up persistent volumes:

# vector-store-snapshot.yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: vector-store-backup-2026-03-17
  namespace: ai-agents
spec:
  volumeSnapshotClassName: csi-snapclass
  source:
    persistentVolumeClaimName: vector-store

Automate snapshots with a CronJob that creates snapshots on a schedule and cleans up old ones.

FAQ

When should I use ReadWriteOnce versus ReadWriteMany for AI agents?

Use ReadWriteOnce (RWO) for single-replica agents with dedicated vector stores or model caches. Use ReadWriteMany (RWX) when multiple agent replicas need to read shared data like a common knowledge base or prompt library. RWX requires an NFS-compatible storage provider like Amazon EFS or Azure Files, which has higher latency than block storage.

How do I expand a PVC without data loss?

If your StorageClass has allowVolumeExpansion: true, edit the PVC and increase spec.resources.requests.storage. Kubernetes expands the volume automatically. For block storage, you may need to restart the Pod for the filesystem to recognize the new size. Always take a VolumeSnapshot before expanding as a safety measure.

Should I store vector embeddings on persistent volumes or in an external database?

For single-node agents processing fewer than one million embeddings, local persistent storage with ChromaDB or FAISS is simpler and lower latency. For multi-replica agents or collections exceeding a few million embeddings, use a managed vector database like Pinecone, Weaviate, or pgvector in PostgreSQL. The external database allows multiple replicas to share the same embedding store and handles replication automatically.

---

#Kubernetes #PersistentStorage #StatefulSets #AIAgents #DataManagement #AgenticAI #LearnAI #AIEngineering