I Built a Self-Hosted RAG Stack That Runs Without Cloud AI APIs

Many teams want AI to answer questions from internal documents, but they cannot send those documents to a cloud service.

Legal teams have confidential case files. Clinics have patient data. Finance teams have client working papers. Manufacturers have proprietary process documentation. Some organizations operate on isolated networks.

For these teams, a cloud-first RAG stack is not an option.

I built LocalRAG to verify a simple idea:

A document question-answering system should be able to run inside a controlled environment, use local models, and return source references for every answer.

The project is open source and the current v0.1.0 release has been verified end to end.

• Repository: https://github.com/Pingredsai/local-rag
• Release: https://github.com/Pingredsai/local-rag/releases/tag/v0.1.0

What LocalRAG does

LocalRAG allows a user to upload documents and ask questions against them.

The verified flow is:

1. Upload a PDF, DOCX, TXT, or Markdown document.
2. Parse and split the document into chunks.
3. Generate embeddings with a local bge-m3 model.
4. Store vectors in Qdrant.
5. Embed the user's question.
6. Retrieve the most relevant chunks.
7. Generate an answer with a local LLM.
8. Return the answer with source references.

The application does not call a cloud AI API at runtime.

Initial installation still requires network access to pull Docker images and Ollama models. After that, the stack can run inside a local or air-gapped environment.

Architecture

LocalRAG uses three main containers:

Browser UI | v FastAPI Backend - document parsing - chunking - RAG orchestration - REST API and static frontend | +--> Qdrant | vector storage and search | +--> Ollama bge-m3 embeddings local LLM generation 

Why these components?

I deliberately did not build a new model runtime, vector database, or agent engine.

The interesting engineering problem is not reinventing Ollama. It is making the complete workflow reproducible, observable, and usable in a private deployment.

Verified test

I tested the current release with:

• Docker Desktop 4.95.0
• Ollama 0.11.3
• Qdrant
• bge-m3 for embeddings
• qwen2.5:3b for generation

The test document was a fictional employee handbook.

The system:

1. Indexed it into three chunks.
2. Received the question: “How many days of annual leave do full-time employees receive?”
3. Retrieved the relevant policy text.
4. Returned “15 days.”
5. Displayed the source document and similarity score in the UI.

A health check returns:

{ "status": "ok", "qdrant": "ok", "ollama": "ok", "llm_model": "qwen2.5:3b", "embed_model": "bge-m3" } 

The repository includes screenshots and a 42-second demo video.

Running it

git clone https://github.com/Pingredsai/local-rag.git cd local-rag cp .env.example .env docker compose up -d docker exec localrag-ollama ollama pull qwen2.5:3b docker exec localrag-ollama ollama pull bge-m3 

Open:

http://localhost:8080 

Four problems I ran into

1. The official Ollama image was too large

The latest Docker image is convenient, but the unpacked image can approach 9 GB. In a slow network environment, that is a serious deployment problem.

I pinned the project to a smaller compatible Ollama release and documented the deployment tradeoff.

This is a reminder that runtime size and download behavior are part of the product experience, not just infrastructure details.

2. A slow Docker registry is not the same as a broken Docker installation

Docker Hub was extremely slow in my environment. The solution was a combination of a mirror, resumable pulls, and local model import in some cases.

I did not hard-code a third-party mirror into the project. Instead, the deployment notes describe the approach so operators can replace it.

3. The original streaming response did not show sources

The first implementation returned text from the streaming endpoint, while sources were only available from the non-streaming endpoint.

That made the UI look like a normal chatbot. The source references, which are essential in a business system, were invisible.

I changed the streaming endpoint to return source metadata in a response header. The frontend reads the header, streams the answer, and renders the source cards when the answer completes.

4. A 1.5B model was not reliable enough for the demo

The smaller model worked on short context but produced inconsistent responses on slightly longer documents.

I switched the demo to qwen2.5:3b and tightened the prompt and generation options.

Local AI is not automatically better just because it is smaller. Model size must match the hardware, use case, and required answer quality.

Current limitations

The first version is intentionally narrow.

It does not include:

• OCR for scanned PDFs
• Multi-tenant permissions
• Multiple isolated knowledge bases
• Audit logging
• A reranker
• Visual agent orchestration

The goal of v0.1.0 is to make the core path dependable:

upload -> index -> retrieve -> answer -> show sources 

I would rather state those boundaries clearly than imply that the project is already a production platform.

What I plan to build next

The long-term goal is to make private AI deployment much simpler and then let users assemble their own local agents.

But I am not going to start by building another general-purpose agent platform.

Existing projects such as Dify, Flowise, Langflow, Open WebUI, AnythingLLM, RAGFlow, and FastGPT already cover a large part of that space.

The missing layer is practical deployment and operations:

• hardware detection
• installation orchestration
• model management
• agent-pack import/export
• permissions and sandboxing
• backup and restore
• offline delivery
• failure diagnostics

My planned sequence is:

1. Validate LocalRAG with real POCs.
2. Build a thin one-click deployment control layer.
3. Define a portable agent.yaml format.
4. Consider a broader platform only after paid usage and repeatable demand.

A platform is not the starting point. It is what remains after repeated delivery.

If your documents cannot leave your network

I am looking for real-world POCs in:

• Legal document search
• Healthcare policies and records
• Finance and audit knowledge bases
• Manufacturing manuals
• Air-gapped or private-network environments

The first POC can be small. We only need to verify:

1. Can your documents be parsed correctly?
2. Does retrieval find the right source passages?
3. Do the answers save useful time?
4. Can the data and permission boundary be satisfied?

Repository:

https://github.com/Pingredsai/local-rag

If you are working on a local-first AI project, I would be happy to compare notes.