Minimum Necessary PHI in AI Prompts: How to Keep LLMs in Bounds

A system prompt that says "do not reveal PHI" is not a HIPAA control. It is a comment in the margin of a contract that the model is free to ignore.

What the rule says

flowchart LR
  Patient["Patient call/chat"] -- "TLS 1.3" --> Edge["Cloudflare WAF"]
  Edge --> App["CallSphere App<br/>HIPAA + SOC 2 aligned"]
  App -- "encrypted" --> AI["AI Voice Agent"]
  AI -- "tool_call · audit" --> Audit[("Audit log<br/>§164.312")]
  AI --> EHR[("EHR · BAA-signed")]
  EHR --> AI
  AI --> Patient

The minimum necessary standard at 45 CFR 164.502(b) and 45 CFR 164.514(d) requires that covered entities and business associates make reasonable efforts to use, disclose, or request only the minimum amount of PHI necessary to accomplish the intended purpose. The rule applies to internal uses, external disclosures, and requests — including, in 2026, every prompt your AI voice agent sends to a large language model.

What it means for AI voice/chat agents

LLMs are designed to consume context. The naive integration pattern — dump the full patient chart into the prompt and let the model figure out what to use — is a textbook minimum necessary violation. An agent scheduling a follow-up only needs the patient identifier, the provider, and the preferred time window. It does not need the full diagnosis history, lab results, or medication list.

The harder lesson is that system prompts are not access controls. The HHS Office for Civil Rights has been clear: instructing an LLM to "not reveal PHI" or "only use the minimum necessary" is not a technical safeguard under 45 CFR 164.312. System prompts can be bypassed by prompt injection, overridden by a model update, or circumvented in multi-step agent workflows. Only data-layer enforcement — where the governance mechanism filters or redacts PHI before it reaches the model — is audit-defensible.

The right architecture wraps every model call with a PHI-aware policy gateway. Inputs are filtered against an allow-list of fields the workflow needs. Outputs are scanned for accidental PHI leakage. The full unredacted record never enters the prompt. Tool calls that need broader context fetch it through a separate authorized path, not through the LLM context window.

CallSphere implementation

CallSphere's voice agents use a structured tool-call pattern instead of full-context prompting. The Healthcare Voice Agent has 12 dedicated tools — eligibility check, appointment search, intake form fill, copay lookup, refill request, prior-authorization status, and others — each with a strict input schema that enforces the minimum necessary fields. The model never sees the full patient chart; it sees only the field-level outputs each tool returns. PHI redaction runs both pre-prompt (inputs are scrubbed against an allow-list) and post-response (outputs are scanned before going back to the caller). Every tool call is logged in our healthcare_voice audit trail with the exact PHI fields requested and returned, so a compliance officer can verify minimum necessary post-hoc, line by line. Start a /trial and you can inspect the audit trail in the dashboard.

Build/audit checklist

1. Inventory every prompt template that touches PHI and list the fields each one currently includes.
2. For every field, write down the workflow purpose — if you cannot, remove it.
3. Move from full-context prompting to strict tool-call schemas with minimum-required fields.
4. Add a pre-prompt PHI redaction layer that strips fields outside the workflow allow-list.
5. Add a post-response PHI scanner that flags accidental leakage before it reaches the caller.
6. Log the exact PHI fields sent and returned on every model call for audit.
7. Document the minimum necessary determination for each workflow in your policy file.
8. Run quarterly red-team tests where a prompt-injection adversary tries to extract additional PHI.
9. Re-review every prompt template when a model is upgraded — behavior shifts can break controls.

FAQ

Does the minimum necessary rule apply to AI prompts? Yes. The rule applies to all uses, disclosures, and requests of PHI, including the act of sending PHI into an LLM prompt for any purpose other than treatment.

Are system prompts a HIPAA control? No. OCR has signaled that system prompts are not technical safeguards. Only data-layer enforcement counts as an auditable control.

How does CallSphere enforce minimum necessary? Strict tool-call schemas with field-level allow-lists, pre-prompt redaction, post-response scanning, and full per-field audit logging.

Can the LLM still hallucinate PHI? It can hallucinate fictitious PHI, which is a clinical safety problem more than a HIPAA problem. Our agents include a hallucination guardrail and only return information from authoritative tool outputs.

Sources

• 45 CFR 164.502(b) Minimum necessary: https://www.law.cornell.edu/cfr/text/45/164.502
• HHS Minimum Necessary Requirement: https://www.hhs.gov/hipaa/for-professionals/privacy/guidance/minimum-necessary-requirement/index.html
• 45 CFR 164.514(d): https://www.ecfr.gov/current/title-45/section-164.514