A self-hosted LLM is an open-weight language model running on hardware you control, answering requests over your own network. Nothing you send it leaves the building. That is the main reason teams ask about it, and it is a good reason.
The second reason people give is cost, and that one is often wrong. I ran my own system's thinking on an 8 GB GPU for months, then moved it to a hosted API, and kept one job at home. Which job stayed, and why, is most of what a small team needs to know before buying a card.
Is a self-hosted LLM worth it?
It depends on the job, not on the team. Three questions settle most cases:
- Must the data stay inside? Contracts, health data, client documents under an NDA. If yes, self-hosting may be the only option, and cost is secondary.
- Is the work steady or bursty? A GPU costs the same idle as busy. Steady, high-volume work uses what you paid for. A few hundred requests a day, arriving in bursts, does not.
- Does the job need judgement? Embeddings, labelling, short summaries: small models do these well. Extracting the right number from a messy document, or deciding what matters: small models do these badly.
Here is what that looked like in my own system, AEGIS. On a replay of 36 real financial emails with known answers, a hosted model extracted 32 correctly. A local 4B model got 13. The local model also averaged about 23 seconds a call against 2 seconds hosted. And the hosted bill for the whole agent fleet came to about five cents a day.
So I moved the judgement work to a hosted API. I did not move the embeddings. The full numbers are in why AEGIS moved off my local GPU; I won't repeat them here.
Hardware: how much VRAM a self-hosted LLM needs
The rule is simple. The model, plus its context, has to fit in GPU memory. If it does not, the server runs part of it on the CPU, and it slows down sharply.
Model size depends on parameter count and quantisation (how many bits store each weight). From Ollama's Llama 3.1 tags page, checked on 1 October 2026:
| Model | Build | Download size |
|---|---|---|
| Llama 3.1 8B | 4-bit (q4_K_M), the default | 4.9 GB |
| Llama 3.1 8B | 8-bit (q8_0) | 8.5 GB |
| Llama 3.1 8B | 16-bit (fp16) | 16 GB |
| Llama 3.1 70B | 4-bit (q4_K_M) | 43 GB |
The context needs memory on top of the weights. Ollama's FAQ says required memory scales with the number of parallel requests times the context length. So four users with long prompts need much more than one.
Ollama also picks a default context from your VRAM. Its context length docs say: under 24 GiB of VRAM, 4k tokens; 24–48 GiB, 32k; 48 GiB or more, 256k. That matters for RAG, below.
My reading of those numbers, as a guide and not a promise:
- 8 GB card: embeddings, and small models (up to about 8B at 4-bit) with short context. This is what I have.
- 24 GB card: 8B-class models with long context, or somewhat larger models at 4-bit.
- 70B-class models: about 43 GB of weights at 4-bit before any context, so two large cards or a workstation-class one.
Check the real split after you load a model. ollama ps shows a PROCESSOR column; you want 100% GPU.
And plan for heat. My GPU runbook fires an alert above 95°C. The first step is to scale the Ollama service to zero and wait until the card is below 70°C before starting it again. A runaway batch of requests can hold a card at 100% long enough to get there.
This is the first part. The full post — including the rest of the working details — is on my site: Self-hosted LLM for a small team: what it takes, and when it pays