Meeting HIPAA and GDPR compliance requirements while running AI inference requires control over every prompt, document, embedding, and log. Both frameworks also place restrictions over who can read that data and how long you can store it.

Self-hosting open-source models on infrastructure you control is the most direct way to manage such requirements, but it’s not the only option.

Agents make the whole situation even more complicated. One user query can turn into a dozen model calls, and every call represents another point where sensitive data can end up.

TL;DR Neither HIPAA nor GDPR requires self-hosting. OpenAI and Anthropic both offer HIPAA-related contracts for eligible features. Self-hosting may be a better option when feature scope, data retention, data location, version pinning, or volume make a hosted contract impractical.

Where Does Sensitive Data Leak in an AI Agent?

Let’s see where an AI agent can expose sensitive data. Is the prompt the only thing that needs protection? Spoiler: its just the tip of the iceberg.

Location

What it holds

Why it matters

Prompts and chat history

User input, often containing health and other personal details

Obvious.

Retrieved context (Retrieval-Augmented Generation, or RAG)

Chunks of records pulled from your index

Often more sensitive than the prompt itself.

Embeddings and vector databases

Numeric versions of your documents

Vec2Text demonstrated exact recovery of 92% of 32-token inputs, including names from clinical notes.

Queues and payload stores

Requests waiting for a GPU

Data at rest is usually not considered.

Logs and traces

Full request bodies when debug logging is on

Ends up in a third-party log tool.

Agent tools and MCP (Model Context Protocol) servers

Whatever the agent sends to them

Each external tool is a new data processor.

Fine-tuned weights and adapters

Patterns from your training data

Models can repeat parts of what they were trained on.

HIPAA vs. GDPR for Inference Layers

Both laws require contracts, appropriate safeguarding, and minimal data. However, they phrase it differently.

Topic

HIPAA (US)

GDPR (EU)

Data covered

Protected Health Information (PHI) held by covered entities and their vendors

Any personal data; health data is a "special category" under Article 9

Vendor contract

Business Associate Agreement (BAA), per 45 CFR 164.502(e) and 164.504(e)

Data Processing Agreement (DPA), per Article 28

Security rules

Technical safeguards in 164.312: access control, audit controls, integrity, authentication, transmission security

"Appropriate" measures under Article 32

Risk assessment

Security risk analysis

Data Protection Impact Assessment (DPIA) under Article 35

Where data can go

No federal location rule

Transfers outside the EU need a legal basis (Chapter V)

End of contract

Vendor must return or destroy PHI

Storage limitation and right to erasure

Do You Have to Self-Host to be Compliant?

No, you don’t need to rely on self-hosting. There are several hosted options that offer protections for sensitive data, but they also come with their own set of limitations.

Hosted option

Contract

The catch

Best for

OpenAI API

BAA through a Business Associate and Healthcare Addendum

Only BAA-eligible endpoints; Zero Data Retention (ZDR) needs separate approval.

Teams that need OpenAI's frontier models.

Anthropic API

BAA for a HIPAA-ready API organization

Batch, Files, Code Execution, and Web Fetch not covered; covered models require 30-day retention and aren’t available with ZDR.

Teams that need Claude for reasoning-heavy steps.

Cloud model platforms (for example AWS)

Covered under the cloud provider's BAA

Coverage depends on each service being on the provider's eligible list.

Teams already under that cloud's BAA.

When Does a Hosted API Still Make Sense?

  • Your volume is low or uneven. Superlinked's self-hosting analysis recommends the API when your monthly inference bill is still under about $2K to $3K.
  • You truly need a frontier model.

When Does Self-Hosting Make Sense?

  • You want to keep your vendor list short. Every additional third party that has access to PHI adds a contract, a potential point of exposure, and another set of terms that can change. If you already have a BAA with your cloud provider, self-hosting in that account adds no new vendor.
  • Your agent needs features outside the BAA scope (batch jobs, file uploads, code execution).
  • Your policy says "no retention," and the covered option keeps data for 30 days.
  • EU data must stay in the EU. There are also state laws, like Texas SB 1188, that limit where records can live.
  • Volume is steady and high.

Which Open-Source Inference Servers Can You Self-Host?

The short, skimmable list:

Tool

Best for

Limitations

vLLM

High-throughput LLM serving with broad model support

One model per server; you build routing and auth around it

SGLang

Multi-turn chat, structured output, RAG with shared prefixes

Smaller ecosystem than vLLM

Ollama

Local development and single-user setups

Not designed for multi-tenant production

TEI (Text Embeddings Inference)

Embeddings and rerankers

One model per server; no generation

TGI (Text Generation Inference)

Existing deployments only

Put in maintenance mode by Hugging Face

SIE (Superlinked Inference Engine)

Many small models (embed, rerank, extract, generate) behind one API

You still run Kubernetes; not built for one huge LLM across many GPUs

For more depth, see Best Inference Servers for Running Open-Source Models in Production and Top Tools for Serving Open-Source Models on Kubernetes.

Your Self-Hosting Compliance Checklist

Contracts and paperwork

  • Sign a BAA (HIPAA) or DPA (GDPR) with your cloud provider, and use only its eligible services.
  • Complete a security risk analysis and, for GDPR, a DPIA.

Network and access

  • Keep inference endpoints on a private network with no public address.
  • Turn on authentication. Some servers accept anonymous requests by default (SIE documents how to enable token auth in its SECURITY.md).
  • Require MFA for anyone who can reach the cluster.

Logs and telemetry

  • Log metadata (model, latency, request ID), not request bodies.
  • Keep an audit trail of who called which model and when.
  • Review or turn off any usage telemetry.

Data and models

  • Redact personal data before it reaches an LLM, a log, or an outside tool.
  • Set retention and deletion rules for vectors, caches, and fine-tuning data.
  • Pin model versions and keep evaluation results for each one.
  • List every tool and MCP server your agent can call.

How Can You Skip Most of This Setup Work?

You may have read the checklist and thought, “Well, that’s a lot!” If so, I’ve got good news: that's where SIE jumps in. (Full disclosure: we’re behind SIE. 👋)

SIE is an Apache 2.0 inference server that runs encoders, rerankers, extractors, and LLMs from one cluster in your own cloud:

If you want help with the security review itself, Native Inference puts Superlinked engineers behind an SIE cluster in your account. Superlinked has no access to the data unless you grant it.

⚠️ Good to know: Superlinked doesn’t currently sign BAAs. For HIPAA workloads, the recommended path is running SIE inside your own infrastructure, with any vendor support kept away from PHI.