Memory Privacy and Isolation: Multi-User Memory Without Data Leakage

The Multi-User Memory Risk

When an AI agent serves multiple users, its memory system becomes a potential vector for data leakage. User A asks the agent about their medical records. User B asks a general question, and the agent accidentally includes details from User A's session in its context. This is not hypothetical — it happens when memory systems lack proper isolation.

Multi-user memory requires strict partitioning, encryption, and access control. No query should ever return memories belonging to a different user, regardless of how similar the content is to the query.

User Isolation Architecture

The foundation is a namespace-per-user design. Each user's memories live in a logically separate partition. The memory store enforces partition boundaries at every access point.

flowchart TD
    MSG(["New message"])
    WORKING["Working memory<br/>rolling window"]
    EPISODIC[("Episodic memory<br/>past sessions")]
    SEMANTIC[("Semantic memory<br/>facts and preferences")]
    SUM["Summarizer<br/>compresses old turns"]
    ROUTER{"Retrieve<br/>needed memories"}
    PROMPT["Assembled context"]
    LLM["LLM"]
    UPD["Memory updater<br/>writes new facts"]
    MSG --> WORKING --> ROUTER
    ROUTER -->|Past sessions| EPISODIC
    ROUTER -->|User facts| SEMANTIC
    EPISODIC --> SUM --> PROMPT
    SEMANTIC --> PROMPT
    WORKING --> PROMPT --> LLM --> UPD
    UPD --> EPISODIC
    UPD --> SEMANTIC
    style ROUTER fill:#4f46e5,stroke:#4338ca,color:#fff
    style LLM fill:#f59e0b,stroke:#d97706,color:#1f2937
    style EPISODIC fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style SEMANTIC fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b

from dataclasses import dataclass, field
from datetime import datetime
from typing import Optional
import hashlib
import secrets

@dataclass
class IsolatedMemory:
    content: str
    user_id: str
    created_at: datetime
    category: str = "general"
    encrypted: bool = False
    id: str = ""

class UserIsolatedMemoryStore:
    def __init__(self):
        # Memories partitioned by user_id
        self._partitions: dict[str, dict[str, IsolatedMemory]] = {}
        self._next_id = 0
        self._encryption_keys: dict[str, bytes] = {}

    def _ensure_partition(self, user_id: str):
        if user_id not in self._partitions:
            self._partitions[user_id] = {}

    def _gen_id(self) -> str:
        self._next_id += 1
        return f"mem_{self._next_id:06d}"

    def add(
        self,
        user_id: str,
        content: str,
        category: str = "general",
    ) -> str:
        self._ensure_partition(user_id)
        mem_id = self._gen_id()
        memory = IsolatedMemory(
            id=mem_id,
            content=content,
            user_id=user_id,
            created_at=datetime.now(),
            category=category,
        )
        self._partitions[user_id][mem_id] = memory
        return mem_id

    def query(
        self,
        user_id: str,
        category: str | None = None,
        keyword: str | None = None,
        top_k: int = 10,
    ) -> list[IsolatedMemory]:
        partition = self._partitions.get(user_id, {})
        results = list(partition.values())

        if category:
            results = [
                m for m in results if m.category == category
            ]
        if keyword:
            results = [
                m for m in results
                if keyword.lower() in m.content.lower()
            ]

        results.sort(key=lambda m: m.created_at, reverse=True)
        return results[:top_k]

The critical design decision here is that every method requires a user_id parameter. There is no method to query across all users. Cross-partition access is architecturally impossible through the public API.

Memory Partitioning Strategies

Beyond the logical namespace approach, you can add physical partitioning for defense in depth.

Database-level isolation uses separate database schemas or tables per user. Even a SQL injection attack cannot cross schema boundaries.

Row-level security uses a single table with a user_id column and database-enforced RLS policies. This is more storage-efficient while still preventing cross-user access at the database layer.

# Example: PostgreSQL row-level security setup
RLS_SETUP_SQL = """
-- Enable RLS on the memories table
ALTER TABLE memories ENABLE ROW LEVEL SECURITY;

-- Policy: users can only access their own rows
CREATE POLICY user_isolation ON memories
    USING (user_id = current_setting('app.current_user_id'));

-- Set user context before queries
SET app.current_user_id = 'user_123';
SELECT * FROM memories;  -- Only returns user_123's rows
"""

Encryption at Rest

Even with partitioning, an attacker who gains database access could read all memories. Encryption at rest adds another layer of protection. Each user gets a unique encryption key, and memory content is encrypted before storage.

from cryptography.fernet import Fernet

class EncryptedMemoryStore(UserIsolatedMemoryStore):
    def _get_user_key(self, user_id: str) -> Fernet:
        if user_id not in self._encryption_keys:
            key = Fernet.generate_key()
            self._encryption_keys[user_id] = key
        return Fernet(self._encryption_keys[user_id])

    def add_encrypted(
        self,
        user_id: str,
        content: str,
        category: str = "general",
    ) -> str:
        fernet = self._get_user_key(user_id)
        encrypted_content = fernet.encrypt(
            content.encode()
        ).decode()

        self._ensure_partition(user_id)
        mem_id = self._gen_id()
        memory = IsolatedMemory(
            id=mem_id,
            content=encrypted_content,
            user_id=user_id,
            created_at=datetime.now(),
            category=category,
            encrypted=True,
        )
        self._partitions[user_id][mem_id] = memory
        return mem_id

    def read_encrypted(
        self, user_id: str, mem_id: str
    ) -> str | None:
        partition = self._partitions.get(user_id, {})
        memory = partition.get(mem_id)
        if not memory:
            return None

        if memory.encrypted:
            fernet = self._get_user_key(user_id)
            return fernet.decrypt(
                memory.content.encode()
            ).decode()
        return memory.content

Access Control Layers

Fine-grained access control goes beyond user isolation. Within a user's partition, different categories of memory may have different sensitivity levels.

from enum import Enum

class SensitivityLevel(Enum):
    PUBLIC = "public"
    PRIVATE = "private"
    SENSITIVE = "sensitive"  # PII, health, financial

ACCESS_POLICIES = {
    SensitivityLevel.PUBLIC: {"agent", "admin", "export"},
    SensitivityLevel.PRIVATE: {"agent", "admin"},
    SensitivityLevel.SENSITIVE: {"admin"},
}

def check_access(
    sensitivity: SensitivityLevel,
    accessor_role: str,
) -> bool:
    allowed = ACCESS_POLICIES.get(sensitivity, set())
    return accessor_role in allowed

def query_with_access_check(
    store: UserIsolatedMemoryStore,
    user_id: str,
    accessor_role: str,
    category: str | None = None,
) -> list[IsolatedMemory]:
    all_memories = store.query(user_id, category=category)
    # Filter based on accessor's permission level
    return [
        m for m in all_memories
        if check_access(
            SensitivityLevel(
                m.category if m.category in {"public", "private", "sensitive"}
                else "private"
            ),
            accessor_role,
        )
    ]

Data Deletion and Right to Erasure

GDPR and similar regulations require the ability to delete all data for a specific user. With partitioned memory, this is straightforward — delete the entire partition.

def delete_user_data(self, user_id: str) -> int:
    partition = self._partitions.pop(user_id, {})
    self._encryption_keys.pop(user_id, None)
    return len(partition)

FAQ

What about shared memories that reference multiple users?

Shared memories should be stored in a separate, non-user-partitioned store with explicit access lists. Never store another user's data inside a user's private partition. Cross-references should use opaque identifiers, never raw content.

How do I handle vector similarity search with encrypted memories?

Encrypted content cannot be embedded or searched directly. The common approach is to store embeddings unencrypted (they do not reveal the original text) but keep the content encrypted. At retrieval time, search embeddings, then decrypt only the returned results.

Is per-user encryption key management too complex?

For production systems, use a key management service (AWS KMS, HashiCorp Vault) instead of generating keys in-process. The KMS handles key rotation, access policies, and audit logging. The code pattern stays the same — you just swap the key source.

---

#MemoryPrivacy #DataIsolation #MultiUser #Security #AgenticAI #LearnAI #AIEngineering