Introduction
A multi-tenant SaaS application may have hundreds or thousands of organizations using the same AI infrastructure.
Each organization has its own documents, policies, customer records, and knowledge base.
A shared RAG pipeline can make this architecture cost-effective, but it also introduces a serious security risk.
Imagine Organization A asks:
"Summarize our customer incidents from last quarter."
The vector database retrieves relevant documents.
One of the results belongs to Organization B.
The LLM receives the document and may incorporate its contents into the response.
The application may have authenticated the user correctly. The database may be encrypted. The REST APIs may have authorization checks.
But the retrieval pipeline has violated tenant isolation.
In this article, we'll build the foundation of a tenant-aware RAG retrieval gateway using Java and Spring Boot.
The focus is not simply adding a tenant ID to a vector record. We want to enforce tenant identity throughout the retrieval lifecycle.
1. Reference architecture
Our implementation will use the following components:
- Java 17.
- Spring Boot.
- PostgreSQL for application metadata.
- A vector database such as Pinecone, Qdrant, or pgvector.
- An embedding model.
- An LLM for response generation.
The architecture:
AUTHENTICATED USER | v SPRING BOOT API | v TENANT RESOLVER | v TENANT CONTEXT | v RAG SERVICE | v VECTOR RETRIEVAL | TENANT FILTER | v RESOURCE AUTHORIZATION | v RERANKER | v CONTEXT BUILDER | v LLM | v RESPONSE VALIDATOR | v USER The important design decision is that the LLM does not determine tenant identity.
Tenant identity comes from the authenticated application context.
2. Project structure
A simple project structure:
tenant-aware-rag/ | +-- controller/ | +-- RagController.java | +-- security/ | +-- TenantContext.java | +-- TenantResolver.java | +-- retrieval/ | +-- VectorSearchService.java | +-- RetrievalFilter.java | +-- RetrievedDocument.java | +-- authorization/ | +-- DocumentAuthorizationService.java | +-- service/ | +-- RagService.java | +-- ContextBuilder.java | +-- cache/ | +-- RagCacheService.java | +-- model/ | +-- RagRequest.java | +-- RagResponse.java | +-- repository/ | +-- DocumentMetadataRepository.java 3. Define the tenant context
First, define the trusted execution context.
public record TenantContext( String tenantId, String userId, Set<String> permissions ) {} The tenant ID must come from authenticated application identity.
Do not accept a tenant ID supplied by the LLM as an authorization input.
For example, a request may contain:
{ "question": "Summarize the incident reports" } The application resolves the tenant using the authenticated principal.
@Component public class TenantResolver { public TenantContext resolve(Authentication authentication) { if (authentication == null || !authentication.isAuthenticated()) { throw new AccessDeniedException( "Authentication required" ); } String userId = authentication.getName(); // Resolve this mapping from a trusted identity/tenant service. String tenantId = resolveTenantForUser(userId); return new TenantContext( tenantId, userId, Set.of("DOCUMENT_READ") ); } private String resolveTenantForUser(String userId) { // Replace with a trusted identity provider or tenant repository. throw new UnsupportedOperationException( "Implement tenant lookup" ); } } This is a design skeleton. In production, the resolver should support users belonging to multiple organizations, tenant membership status, and permission changes.
4. Store tenant metadata with every chunk
A document may be split into hundreds of chunks.
Every chunk needs tenant ownership and document-level metadata.
Example vector record:
{ "id": "chunk-1001", "values": [0.12, 0.34, 0.56], "metadata": { "tenant_id": "tenant-a", "document_id": "doc-1001", "classification": "internal", "source": "incident-report.pdf", "chunk_number": 1 } } Do not store tenant ownership only on the original document.
When chunks become independently retrievable objects, ownership and authorization metadata must be available at the chunk level.
Also validate tenant ownership during ingestion. Never trust tenant metadata extracted from a document or supplied by a client.
5. Define the retrieval contract
Create a normalized retrieval request:
public record RetrievalRequest( String tenantId, String userId, List<String> permittedDocumentIds, String query, int topK ) {} The tenantId and permissions should come from the trusted execution context.
The model must not be able to override them.
Define the result:
public record RetrievedDocument( String chunkId, String documentId, String tenantId, String content, double score ) {} And the service contract:
public interface VectorSearchService { List<RetrievedDocument> search( RetrievalRequest request ); } 6. Enforce tenant filters during vector retrieval
The exact query syntax depends on the vector database.
Conceptually, the query should look like:
{ "vector": [0.12, 0.34, 0.56], "topK": 10, "filter": { "tenant_id": { "$eq": "tenant-a" } } } For a database supporting SQL-based vector search, the equivalent idea might be:
SELECT chunk_id, document_id, tenant_id, content, embedding <=> :query_embedding AS distance FROM document_chunks WHERE tenant_id = :tenant_id ORDER BY embedding <=> :query_embedding LIMIT :top_k; The tenant filter must be applied as part of candidate selection.
Do not perform global vector search and rely exclusively on filtering results afterward.
A production implementation must also validate document-level permissions. Tenant equality does not automatically mean every user within that tenant can access every document.
7. Add a second authorization check
Tenant filtering protects organizational boundaries.
However, users within an organization may have different permissions.
For example:
- Finance users can access financial documents.
- HR users can access HR documents.
- Support users can access support documentation.
Define an authorization service:
public interface DocumentAuthorizationService { boolean canRead( String userId, String tenantId, String documentId ); } Validate the documents returned by vector search:
@Service public class RetrievalAuthorizationService { private final DocumentAuthorizationService authorization; public RetrievalAuthorizationService( DocumentAuthorizationService authorization) { this.authorization = authorization; } public List<RetrievedDocument> filterAuthorized( TenantContext context, List<RetrievedDocument> documents) { return documents.stream() .filter(document -> context.tenantId().equals(document.tenantId()) ) .filter(document -> authorization.canRead( context.userId(), context.tenantId(), document.documentId() ) ) .toList(); } } This second check is defense in depth.
The vector database filter limits the candidate set, while the authorization service verifies the user's actual resource permissions.
The check should use current authorization data, not permissions inferred by the LLM.
8. Build the RAG service
Now combine the retrieval and authorization steps.
@Service public class RagService { private final VectorSearchService vectorSearch; private final RetrievalAuthorizationService authorization; private final EmbeddingService embeddingService; private final LlmService llmService; public RagService( VectorSearchService vectorSearch, RetrievalAuthorizationService authorization, EmbeddingService embeddingService, LlmService llmService) { this.vectorSearch = vectorSearch; this.authorization = authorization; this.embeddingService = embeddingService; this.llmService = llmService; } public String answer( TenantContext context, String question) { float[] embedding = embeddingService.embed(question); RetrievalRequest request = new RetrievalRequest( context.tenantId(), context.userId(), List.of(), question, 10 ); List<RetrievedDocument> candidates = vectorSearch.search(request); List<RetrievedDocument> authorized = authorization.filterAuthorized( context, candidates ); String prompt = buildContext( question, authorized ); return llmService.generate(prompt); } } This is an illustrative service skeleton. The embedding is shown to establish the flow; the vector adapter must actually use it in its search request.
The context builder should include only authorized documents.
If a document fails authorization, it must never be sent to the reranker or LLM.
9. Prevent semantic cache leakage
Caching is another area where multi-tenant RAG systems can fail.
Suppose two tenants ask semantically similar questions.
If the cache uses only the normalized question, the second tenant may receive an answer generated from the first tenant's documents.
A cache key should include the effective authorization context.
For example:
public String buildCacheKey( TenantContext context, String normalizedQuery, String modelVersion, String retrievalVersion) { String rawKey = String.join( "|", context.tenantId(), context.userId(), String.join(",", context.permissions().stream().sorted().toList()), normalizedQuery, modelVersion, retrievalVersion ); return sha256(rawKey); } This is a starting point. For a production system, represent the effective authorization scope using a stable permission-scope identifier or version rather than relying only on a simple permission list.
Also consider:
- Permission changes.
- Document updates and deletion.
- Tenant membership changes.
- Model and prompt versions.
- Retrieval configuration changes.
- Cache expiration and invalidation.
Never reuse a cached answer without confirming that the current caller is still authorized to access its underlying source information.
10. Add integration tests for tenant isolation
This is where the architecture becomes meaningful.
Create two tenants:
Tenant A document-a1 document-a2 Tenant B document-b1 document-b2 Now test the retrieval boundaries.
@Test void shouldNeverReturnDocumentsFromAnotherTenant() { TenantContext tenantA = new TenantContext( "tenant-a", "user-a", Set.of("DOCUMENT_READ") ); List<RetrievedDocument> documents = List.of( new RetrievedDocument( "chunk-a1", "document-a1", "tenant-a", "Tenant A incident report", 0.91 ), new RetrievedDocument( "chunk-b1", "document-b1", "tenant-b", "Tenant B confidential report", 0.95 ) ); List<RetrievedDocument> result = authorization.filterAuthorized( tenantA, documents ); assertTrue( result.stream().allMatch( doc -> doc.tenantId().equals("tenant-a") ) ); } This tests the authorization filter, but it is not enough.
Add integration tests that inspect the actual vector query and verify that the tenant filter is applied before candidate retrieval.
Also test:
- Missing tenant context.
- Forged tenant IDs in user input.
- Unauthorized documents within the same tenant.
- Cross-tenant semantic cache hits.
- Permission revocation after a cache entry is created.
- Unauthorized document IDs reaching the reranker.
- Unauthorized content reaching the LLM.
- Cross-tenant citations in the final response.
The most important assertion is:
No unauthorized document should enter the model's context window.
11. Production hardening
Before deploying this architecture, address several additional concerns.
Ingestion security
- Derive tenant ownership from trusted application identity.
- Validate ownership before processing files.
- Preserve tenant metadata through chunking.
- Prevent document ID collisions.
- Securely delete all associated chunks when documents are removed.
Retrieval security
- Apply filters during candidate generation.
- Validate document-level permissions.
- Restrict access to vector database credentials.
- Avoid exposing raw vector queries to the LLM.
- Ensure reranking receives only authorized candidates.
Cache security
- Scope cache entries to effective authorization context.
- Invalidate after permission changes.
- Avoid caching sensitive responses globally.
- Include retrieval and model versions where relevant.
Operational security
- Record retrieval and authorization decisions.
- Monitor cross-tenant access attempts.
- Avoid logging confidential document content.
- Fail closed when tenant context or authorization services are unavailable.
- Test isolation continuously.
Conclusion
Multi-tenant RAG is not simply a vector database configuration problem.
It is an end-to-end authorization problem.
A secure implementation must preserve tenant identity from ingestion through retrieval, reranking, caching, and final generation.
Traditional application security protects the API.
Enterprise AI security must also protect the information entering the model's context.
The context window is part of the security boundary.
If unauthorized information reaches it, the architecture has already failed.
About the Author
I'm Ashwin Giridharan, a Cloud & Enterprise Solutions Architect focused on building scalable systems and exploring the architecture of AI.
My interests include enterprise architecture, cloud platforms, distributed systems, Generative AI, LLMs, RAG, Agentic AI, and taking AI solutions from experimentation to production.
I share practical insights, technical explorations, and lessons learned while building and understanding modern technology systems.
Connect and explore my work:
Website: https://ashwingiridharan.com/