Deploying LLMs Locally: Ollama, llama.cpp, and vLLM Comparison
Run any model on your hardware — no API keys, no cloud, complete privacy.
Introduction
Local LLMs have matured. You can now run 7B-70B models on consumer hardware with near-cloud performance. This guide covers the three main engines: Ollama (easiest), llama.cpp (most flexible), vLLM (highest throughput).
Quick Comparison
| Feature | Ollama | llama.cpp | vLLM |
|---|---|---|---|
| Ease of use | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐ |
| Model formats | GGUF | GGUF | HF/Safetensors |
| API compatible | ✅ OpenAI | ✅ OpenAI | ✅ OpenAI |
| Multimodal | ✅ | ✅ | ✅ |
| Batching | ❌ | ❌ | ✅ PagedAttention |
| Quantization | Auto | Manual | Auto (AWQ/GPTQ) |
| GPU/CPU | Both | Both | GPU (CUDA/ROCm) |
| Best for | Devs, local chat | Embedded, edge | Production serving |
Hardware Requirements
| Model | Quantization | VRAM/RAM | CPU Only |
|---|---|---|---|
| 7B | Q4_K_M | 6 GB | ✅ Slow |
| 7B | Q8_0 | 8 GB | ✅ Usable |
| 13B | Q4_K_M | 10 GB | ✅ Slow |
| 33B | Q4_K_M | 20 GB | ❌ |
| 70B | Q4_K_M | 40 GB | ❌ |
Rule of thumb: VRAM ≈ params × 0.5 (Q4) to 1.0 (Q8) in GB.
Option 1: Ollama — Easiest Start
Install
# Linux curl -fsSL https://ollama.com/install.sh | sh # macOS brew install ollama # Windows winget install Ollama.Ollama Run a Model
# Pull and run (auto-downloads GGUF) ollama run mistral:7b-instruct-q4_K_M # List local models ollama list # Show model info ollama show mistral:7b-instruct-q4_K_M OpenAI-Compatible API
# Start server (default port 11434) ollama serve # Test API curl -X POST http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "mistral:7b-instruct-q4_K_M", "messages": [{"role": "user", "content": "Hello!"}], "stream": false }' Python Client
import requests def chat(messages, model="mistral:7b-instruct-q4_K_M"): response = requests.post( "http://localhost:11434/v1/chat/completions", json={"model": model, "messages": messages, "stream": False} ) return response.json()["choices"][0]["message"]["content"] print(chat([{"role": "user", "content": "Write a haiku about GPUs"}])) Custom Models (Modelfile)
# Modelfile FROM ./my-model.Q4_K_M.gguf TEMPLATE "{{ .System }}\n\n{{ .Prompt }}" PARAMETER temperature 0.7 PARAMETER top_p 0.9 PARAMETER num_ctx 8192 SYSTEM "You are a helpful coding assistant." ollama create my-coder -f Modelfile ollama run my-coder Option 2: llama.cpp — Maximum Control
Build
git clone https://github.com/ggerganov/llama.cpp cd llama.cpp make LLAMA_CUBLAS=1 # For NVIDIA GPU # or: make LLAMA_METAL=1 # For Apple Silicon Convert HF → GGUF
# Download model (needs HF token for gated models) huggingface-cli download meta-llama/Llama-3.1-8B-Instruct --local-dir ./llama-3.1-8b # Convert python convert-hf-to-gguf.py ./llama-3.1-8b \ --outfile ./llama-3.1-8b.Q4_K_M.gguf \ --outtype q4_k_m Quantization Types
| Type | Size (7B) | Quality | Speed |
|---|---|---|---|
| Q2_K | ~2.8 GB | Low | Fastest |
| Q3_K_M | ~3.3 GB | Medium | Fast |
| Q4_K_M | ~3.8 GB | Good | Balanced |
| Q5_K_M | ~4.7 GB | Better | Slower |
| Q6_K | ~5.3 GB | High | Slower |
| Q8_0 | ~7.2 GB | Near fp16 | Slowest |
Run Server (OpenAI Compatible)
# Basic server ./llama-server -m ./model.Q4_K_M.gguf -c 8192 --port 8080 # With GPU offload (all layers) ./llama-server -m ./model.Q4_K_M.gguf -c 8192 --port 8080 -ngl 99 # With flash attention (faster) ./llama-server -m ./model.Q4_K_M.gguf -c 8192 --port 8080 -ngl 99 -fa Python Client
from llama_cpp import Llama llm = Llama( model_path="./model.Q4_K_M.gguf", n_ctx=8192, n_gpu_layers=-1, # -1 = all layers on GPU verbose=False ) # Chat completion response = llm.create_chat_completion( messages=[{"role": "user", "content": "Hello!"}], temperature=0.7, max_tokens=512 ) print(response["choices"][0]["message"]["content"]) # Streaming for chunk in llm.create_chat_completion( messages=[{"role": "user", "content": "Write a story"}], stream=True ): delta = chunk["choices"][0]["delta"] if "content" in delta: print(delta["content"], end="", flush=True) Advanced: Speculative Decoding
# Draft model (small) + target model (large) ./llama-server -m ./llama-3.1-8b.Q4_K_M.gguf \ -md ./llama-3.1-1b.Q4_K_M.gguf \ -c 8192 --port 8080 -ngl 99 Option 3: vLLM — Production Throughput
Install
# Requires CUDA 12.1+ and Python 3.10+ pip install vllm # For ROCm (AMD) pip install vllm --extra-index-url https://download.pytorch.org/whl/rocm6.0 Run Server
# HF model (auto-downloads) vllm serve meta-llama/Llama-3.1-8B-Instruct \ --tensor-parallel-size 1 \ --gpu-memory-utilization 0.9 \ --max-model-len 8192 \ --dtype auto \ --api-key your-secret-key Quantized Models (AWQ/GPTQ)
# AWQ (faster inference) vllm serve TheBloke/Llama-3.1-8B-Instruct-AWQ \ --quantization awq \ --gpu-memory-utilization 0.9 # GPTQ vllm serve TheBloke/Llama-3.1-8B-Instruct-GPTQ \ --quantization gptq \ --gpu-memory-utilization 0.9 Multi-GPU (Tensor Parallel)
# 2 GPUs vllm serve meta-llama/Llama-3.1-70B-Instruct \ --tensor-parallel-size 2 \ --gpu-memory-utilization 0.9 \ --max-model-len 4096 Python Client (OpenAI Compatible)
from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="your-secret-key" ) # Chat response = client.chat.completions.create( model="meta-llama/Llama-3.1-8B-Instruct", messages=[{"role": "user", "content": "Hello!"}], temperature=0.7, max_tokens=512 ) print(response.choices[0].message.content) # Streaming stream = client.chat.completions.create( model="meta-llama/Llama-3.1-8B-Instruct", messages=[{"role": "user", "content": "Write a poem"}], stream=True ) for chunk in stream: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="", flush=True) Benchmarks (vLLM vs Others)
Model: Llama-3.1-8B, Batch: 32, Context: 2048 vLLM (PagedAttention): ~450 tok/s llama.cpp (batch 32): ~280 tok/s Ollama (no batching): ~120 tok/s Model Recommendations (Oct 2024)
General Purpose
| Size | Model | Best For |
|---|---|---|
| 1B | SmolLM2-1.7B-Instruct | Edge, mobile |
| 3B | Phi-3.5-mini-instruct | Fast, capable |
| 7B | Mistral-Nemo-12B-Instruct | Best 7B-class |
| 7B | Llama-3.1-8B-Instruct | Standard baseline |
| 7B | Qwen2.5-7B-Instruct | Multilingual, code |
| 13B | Nemotron-3-Ultra | Reasoning |
Coding
| Size | Model |
|---|---|
| 7B | CodeQwen1.5-7B |
| 7B | DeepSeek-Coder-V2-Lite |
| 13B | CodeLlama-13B-Instruct |
| 33B | DeepSeek-Coder-V2-33B |
Reasoning
| Size | Model |
|---|---|
| 7B | Nemotron-3-Ultra |
| 13B | QwQ-32B-Preview (distilled) |
| 33B | Nemotron-3-Ultra-33B |
Performance Tuning
llama.cpp
# Optimal thread count (physical cores) ./llama-server -m model.gguf -t $(nproc --all) -ngl 99 # Batch size for throughput ./llama-server -m model.gguf -b 512 -ub 512 # Flash attention (requires Hopper/Ada or Apple Silicon) ./llama-server -m model.gguf -fa vLLM
# Increase batch size vllm serve model --max-num-batched-tokens 8192 --max-num-seqs 256 # Chunked prefill (long contexts) vllm serve model --enable-chunked-prefill --max-model-len 32768 # Prefix caching (shared prompts) vllm serve model --enable-prefix-caching Ollama
# Environment variables OLLAMA_NUM_PARALLEL=4 # Concurrent requests OLLAMA_MAX_LOADED_MODELS=2 # Keep models in VRAM OLLAMA_FLASH_ATTENTION=1 # Enable flash attention Cost Comparison (Monthly)
| Setup | Hardware | Models | Est. Cost |
|---|---|---|---|
| Ollama + RTX 3090 | $800 used | 7B-33B Q4 | $15/mo electricity |
| llama.cpp + RTX 4090 | $1600 | 7B-70B Q4 | $25/mo electricity |
| vLLM + 2×A100 80GB | $30k | 70B fp8 | $500/mo colocation |
| Cloud API (OpenRouter) | — | Any | $50-500/mo usage |
Break-even: Local GPU pays for itself in 6-18 months vs cloud APIs.
Common Issues
| Problem | Fix |
|---|---|
| OOM on GPU | Reduce n_gpu_layers, use smaller quantization, offload to CPU |
| Slow CPU inference | Use Q4_K_M, enable BLAS (LLAMA_BLAS=1), reduce context |
| Model not loading | Check GGUF compatibility, try llama.cpp convert script |
| API connection refused | Check port, firewall, OLLAMA_HOST=0.0.0.0 |
| Poor quality | Increase temperature, try larger model, better quantization |
Decision Guide
START: What's your use case? │ ├─→ "Just chat / quick tests" → **Ollama** │ ├─→ "Embed in app / edge device / custom quant" → **llama.cpp** │ ├─→ "Serve many users / high throughput / production" → **vLLM** │ └─→ "Apple Silicon Mac" → **Ollama** or **llama.cpp (Metal)** Conclusion
Local LLMs are production-ready. Choose:
- Ollama for developer experience
- llama.cpp for control and edge deployment
- vLLM for serving scale
All three expose OpenAI-compatible APIs — swap providers without code changes.
Next: Add RAG (see previous tutorial), function calling, or fine-tune your own LoRA.
Part of the Local LLM series on Dev.to
