Secret Management for AI Agents: Vault, AWS Secrets Manager, and Rotation Policies

The Secret Management Problem in Agentic Systems

AI agents need credentials to function: API keys for LLM providers, database passwords, OAuth tokens for third-party services, and encryption keys for secure communication. The way these secrets are stored and accessed directly determines the security posture of your entire agent system.

Hardcoded secrets in configuration files, environment variables baked into container images, or secrets stored in plaintext databases are common mistakes that create catastrophic attack surfaces. When an agent is compromised, every secret it can access is compromised. The goal of secret management is to minimize what each agent can access, ensure credentials rotate frequently, and maintain a complete audit trail.

HashiCorp Vault for Agent Secrets

Vault provides centralized secret storage with fine-grained access control, automatic rotation, and comprehensive audit logging. Each agent authenticates to Vault with its own identity and receives only the secrets its policy permits:

flowchart LR
    INPUT(["User intent"])
    PARSE["Parse plus<br/>classify"]
    PLAN["Plan and tool<br/>selection"]
    AGENT["Agent loop<br/>LLM plus tools"]
    GUARD{"Guardrails<br/>and policy"}
    EXEC["Execute and<br/>verify result"]
    OBS[("Trace and metrics")]
    OUT(["Outcome plus<br/>next action"])
    INPUT --> PARSE --> PLAN --> AGENT --> GUARD
    GUARD -->|Pass| EXEC --> OUT
    GUARD -->|Fail| AGENT
    AGENT --> OBS
    style AGENT fill:#4f46e5,stroke:#4338ca,color:#fff
    style GUARD fill:#f59e0b,stroke:#d97706,color:#1f2937
    style OBS fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style OUT fill:#059669,stroke:#047857,color:#fff

import hvac
from functools import lru_cache

class AgentVaultClient:
    """Manages agent access to HashiCorp Vault."""

    def __init__(self, vault_url: str, auth_method: str = "kubernetes"):
        self.client = hvac.Client(url=vault_url)
        self.auth_method = auth_method

    def authenticate_kubernetes(self, role: str) -> None:
        """Authenticate using Kubernetes service account token."""
        with open(
            "/var/run/secrets/kubernetes.io/serviceaccount/token"
        ) as f:
            jwt = f.read()

        self.client.auth.kubernetes.login(role=role, jwt=jwt)

    def get_secret(self, path: str) -> dict:
        """Retrieve a secret from Vault's KV v2 engine."""
        response = self.client.secrets.kv.v2.read_secret_version(
            path=path,
            mount_point="secret",
        )
        return response["data"]["data"]

    def get_dynamic_database_credentials(self, role: str) -> dict:
        """Get short-lived database credentials from Vault."""
        response = self.client.secrets.database.generate_credentials(
            name=role
        )
        return {
            "username": response["data"]["username"],
            "password": response["data"]["password"],
            "ttl": response["lease_duration"],
            "lease_id": response["lease_id"],
        }

    def renew_lease(self, lease_id: str) -> None:
        """Renew a dynamic secret lease before it expires."""
        self.client.sys.renew_lease(lease_id=lease_id)

# Agent startup
vault = AgentVaultClient("https://vault.internal:8200")
vault.authenticate_kubernetes(role="research-agent")

# Get LLM API key
llm_secrets = vault.get_secret("agents/research/llm")
api_key = llm_secrets["openai_api_key"]

# Get dynamic database credentials (auto-expire after TTL)
db_creds = vault.get_dynamic_database_credentials("research-agent-readonly")

Vault Policies for Agents

Write policies that follow least privilege. Each agent role gets a policy that grants access only to its required secrets:

# Vault policy as HCL (stored in Vault, shown here for reference)
RESEARCH_AGENT_POLICY = """
# Read-only access to LLM API keys
path "secret/data/agents/research/llm" {
  capabilities = ["read"]
}

# Dynamic database credentials (read-only role)
path "database/creds/research-agent-readonly" {
  capabilities = ["read"]
}

# Deny access to other agents' secrets
path "secret/data/agents/admin/*" {
  capabilities = ["deny"]
}

# Deny access to infrastructure secrets
path "secret/data/infrastructure/*" {
  capabilities = ["deny"]
}
"""

def setup_vault_policy(client: hvac.Client, agent_role: str) -> None:
    """Create or update a Vault policy for an agent role."""
    policy_map = {
        "research-agent": RESEARCH_AGENT_POLICY,
    }

    policy = policy_map.get(agent_role)
    if policy:
        client.sys.create_or_update_policy(
            name=f"{agent_role}-policy",
            policy=policy,
        )

AWS Secrets Manager with Automatic Rotation

For AWS-hosted agents, Secrets Manager provides native rotation with Lambda functions. Build a client that caches secrets locally and refreshes them on rotation:

import boto3
import json
from datetime import datetime, timedelta

class AWSAgentSecretManager:
    """Manages agent secrets via AWS Secrets Manager with caching."""

    def __init__(self, region: str = "us-east-1"):
        self.client = boto3.client("secretsmanager", region_name=region)
        self._cache: dict[str, tuple[dict, datetime]] = {}
        self._cache_ttl = timedelta(minutes=5)

    def get_secret(self, secret_name: str) -> dict:
        """Retrieve a secret with local caching."""
        now = datetime.utcnow()

        if secret_name in self._cache:
            value, cached_at = self._cache[secret_name]
            if now - cached_at < self._cache_ttl:
                return value

        response = self.client.get_secret_value(SecretId=secret_name)
        secret_data = json.loads(response["SecretString"])

        self._cache[secret_name] = (secret_data, now)
        return secret_data

    def create_agent_secret(
        self, agent_id: str, secrets: dict, rotation_days: int = 30
    ) -> str:
        """Create a new secret for an agent with rotation enabled."""
        secret_name = f"agents/{agent_id}/credentials"

        response = self.client.create_secret(
            Name=secret_name,
            SecretString=json.dumps(secrets),
            Tags=[
                {"Key": "agent_id", "Value": agent_id},
                {"Key": "managed_by", "Value": "agent-platform"},
            ],
        )

        # Enable automatic rotation
        self.client.rotate_secret(
            SecretId=secret_name,
            RotationLambdaARN=(
                "arn:aws:lambda:us-east-1:123456:function:agent-secret-rotator"
            ),
            RotationRules={"AutomaticallyAfterDays": rotation_days},
        )

        return response["ARN"]

    def invalidate_cache(self, secret_name: str) -> None:
        """Force refresh on next access."""
        self._cache.pop(secret_name, None)

Agent Access Patterns

Design your agent's secret access to minimize exposure. Load secrets at startup, never log them, and clear them from memory when no longer needed:

import os
import ctypes

def secure_clear(secret: str) -> None:
    """Overwrite a string's memory buffer to prevent lingering secrets."""
    if not secret:
        return
    buf = ctypes.create_string_buffer(len(secret))
    ctypes.memmove(id(secret) + 49, buf, len(secret))

class AgentSecretContext:
    """Context manager that loads and securely clears secrets."""

    def __init__(self, secret_manager: AWSAgentSecretManager, secret_name: str):
        self.manager = secret_manager
        self.secret_name = secret_name
        self.secrets: dict = {}

    def __enter__(self) -> dict:
        self.secrets = self.manager.get_secret(self.secret_name)
        return self.secrets

    def __exit__(self, exc_type, exc_val, exc_tb) -> None:
        # Clear secrets from memory
        for key in list(self.secrets.keys()):
            if isinstance(self.secrets[key], str):
                secure_clear(self.secrets[key])
            self.secrets[key] = None
        self.secrets.clear()

# Usage
sm = AWSAgentSecretManager()
with AgentSecretContext(sm, "agents/research/llm") as creds:
    api_key = creds["api_key"]
    # Use api_key for LLM calls within this block
# Secrets cleared from memory after block exits

FAQ

Should each agent instance have its own credentials?

Yes, whenever possible. Unique credentials per agent instance enable precise audit trails — you can trace any action to a specific agent. Dynamic secrets from Vault are ideal for this because each agent instance gets unique, short-lived credentials without manual provisioning.

How do I handle secret rotation without agent downtime?

Use the dual-secret pattern: when rotating, both the old and new credentials are valid during a grace period. The secret manager returns the new credential, but the old one remains active until the grace period expires. Agents that cache secrets locally refresh on the next cache miss, picking up the new credential seamlessly.

What happens if Vault or Secrets Manager is unreachable?

Build resilience with local caching and graceful degradation. Cache the last known valid credentials with a bounded TTL. If the secret manager is unreachable and the cache TTL has expired, the agent should fail safely rather than continue with stale credentials. Implement health checks that alert when the secret manager connection is degraded.

---

#SecretManagement #HashiCorpVault #AWSSecretsManager #CredentialRotation #AISecurity #AgenticAI #LearnAI #AIEngineering