RAG Privacy: Indexing Sensitive Data Without Leaking

The Privacy Surface

RAG indexes documents. Documents contain sensitive data. The vector index is searchable; the retrieved chunks flow to LLM providers; outputs may contain sensitive content. Each step is a potential leak.

By 2026 the patterns for privacy-respecting RAG are well-understood. This piece walks through the layered defense.

The Threat Model

flowchart TB
 Threats[Threats] --> T1[Cross-tenant leak]
 Threats --> T2[Cross-user leak]
 Threats --> T3[Provider data exposure]
 Threats --> T4[Embedding inversion]
 Threats --> T5[Prompt-injection exfiltration]

Five distinct threats. Each has its own mitigations.

Cross-Tenant Leak

Documents from one tenant's corpus retrieved for another tenant's query.

Defenses:

• Per-tenant indexes (separate physical stores)
• Per-tenant collections within shared stores
• Mandatory tenant filter on every query
• Audit log of retrievals with tenant attribution
• Multi-tenant isolation tests in CI

Cross-User Leak

Within a tenant, documents user A should not see retrieved for user B's query.

Defenses:

• Document-level ACLs at index time
• ACL filter on every retrieval (filter by user permissions)
• Re-validate ACL at generation time (do not trust the retriever alone)

Provider Data Exposure

Retrieved chunks sent to an LLM provider may be processed, logged, or used for training.

Defenses:

• BAA / DPA with the provider
• "Do not use for training" terms in the contract
• On-prem inference for the most sensitive workloads
• Redact obvious PII before sending

Embedding Inversion

In theory, an attacker with embeddings can partially reconstruct the original text.

Defenses:

• Treat embeddings as sensitive (encrypt at rest, restrict access)
• For ultra-sensitive corpora, consider differential-privacy-aware embedding
• Audit who has access to the vector store

In practice for 2026, full inversion is not yet a practical attack on most embeddings. But access controls should still treat the embedding store as sensitive.

Prompt-Injection Exfiltration

A malicious document in the corpus contains instructions to "exfiltrate the previous context."

Defenses:

• Treat all retrieved content as untrusted
• System prompt forbids instruction-following on retrieved content
• Output guards detect exfiltration patterns
• Tool-permission scoping limits damage if exfiltration occurs

A Layered Architecture

flowchart LR
 Doc[Document with PII] --> Redact[PII redaction at index]
 Redact --> Embed[Embed with model]
 Embed --> Store[Per-tenant vector store]
 Query[User query + ACL] --> Retrieve[Filter by tenant + ACL]
 Retrieve --> Context[Filtered context]
 Context --> LLM[LLM via BAA provider]
 LLM --> Output[Output via guard]

Each layer adds a check. Compromise of one layer does not compromise the system.

Indexing Patterns

When indexing sensitive data:

• Classify documents at ingest (PII, PHI, financial, internal-only, public)
• Apply redaction or full-document gating per classification
• Tag chunks with classification
• Index ACLs alongside content

Access Control at Retrieval

flowchart TD
 Q[Query] --> User[Resolve user identity]
 User --> Perms[Resolve permissions]
 Perms --> Filter[Filter retrievable docs]
 Filter --> Top[Top K from filtered set]

The filter is non-negotiable. Without it, the retriever returns everything and you trust the LLM not to use what it shouldn't — a bad bet.

Output Filtering

Even with everything else right, the output may include sensitive content the model inferred. Patterns:

• PII detection on output
• Redaction or refusal on detection
• Watermarking outputs in regulated contexts

Audit

Every retrieval should be logged with:

• User identity
• Tenant
• Query
• Documents returned
• Time
• Tool / endpoint

Compliance reviews depend on this log. Without it, breach analysis is guesswork.

Compliance Mapping

Standard	RAG-relevant requirement
HIPAA	BAA with provider, PHI redaction at boundaries
GDPR	DSAR, right-to-be-forgotten support, EU residency
	Access logs, encryption at rest and transit
PCI DSS	No card data in indexes

A privacy-first RAG architecture often satisfies multiple frameworks at once.

What CallSphere Does

For our healthcare voice agent's RAG:

• Per-tenant vector stores
• HIPAA BAA with the LLM provider
• PII redaction in transcripts before indexing
• ACL filter at retrieval (user can only retrieve their own appointments, etc.)
• Audit log of every retrieval
• Output guard for PII

Layered defenses; no single failure compromises the system.

Sources

• "Privacy in RAG" research — https://arxiv.org
• HIPAA Privacy Rule — https://www.hhs.gov/hipaa
• "GDPR for AI" guidance — https://gdpr.eu
• "Differential privacy in embeddings" — https://arxiv.org
• "Access control patterns" Pinecone — https://www.pinecone.io/learn