I've stood up both of these engines on the same machine more than once, and the question I keep getting is simply: which one do I install? The honest answer has almost nothing to do with benchmark screenshots and everything to do with how many requests land on your endpoint at the same second, and what hardware you actually own.

Below is the decision path I use, followed by the commands that get either engine behind an OpenAI-compatible /v1 API. Budget around 45 minutes for one engine end-to-end, or roughly 75 if you want to run both and compare. It's intermediate territory — you should be comfortable with a terminal, cmake, and a Python virtualenv.

What actually separates the two engines

llama.cpp is a C/C++ inference engine that runs GGUF files. GGUF is a single-file model format holding the weights, usually compressed to 4-bit, which means a model fits into far less memory than its full-precision version. It runs on CPU-only machines, CUDA, Metal, ROCm and Vulkan.

vLLM is a Python serving engine that loads raw Hugging Face weights. Its two defining pieces are PagedAttention, a KV-cache memory manager, and continuous batching — new requests join a batch that is already running instead of waiting for the current one to finish.

llama.cpp vLLM
Model format GGUF, quantized (e.g. model.Q4_K_M.gguf) Hugging Face safetensors, unquantized or supported quantized
Hardware CPU-only, CUDA, Metal, ROCm, Vulkan CUDA GPUs; multi-GPU via --tensor-parallel-size
Concurrency One process with a small number of parallel slots (-np) Continuous batching + PagedAttention
Choose it when One developer on a laptop or desktop, limited VRAM, or no GPU at all Many concurrent requests on a server GPU

Three questions that decide it

Answer these in writing before you touch a package manager.

  1. How many requests arrive in the same second? More than one, and you want vLLM. A single chat session does not need continuous batching.
  2. Does the model fit in VRAM at full precision? Compare the parameter count against what nvidia-smi reports. If it doesn't fit, going the quantization route with llama.cpp is the shorter path.
  3. Is there a GPU at all? With no discrete GPU, llama.cpp is the only option of the two.

A concrete example from my own habits: a single 24 GB card running one chat session gets llama.cpp with a Q4_K_M GGUF. The same card serving a team of 20 people calling an API gets vLLM. Concurrency decides the engine; quantization is the fallback when the model refuses to fit.

Check the machine before installing anything

Both engines fail late and loudly — missing drivers and occupied ports surface as confusing errors after a long build or model download. So verify first.

What you need: Linux (x86_64 or aarch64) or macOS. On Windows, use WSL2 and keep the project inside the WSL filesystem, not on a mounted C:\ path. For llama.cpp you need git, cmake, make and g++. For vLLM, a Python 3 virtualenv — never the system Python. Plus a .gguf file or a Hugging Face repository ID, and free ports 8080 and 8000. Exact supported Python and CUDA versions shift between releases, so check each project's docs.

nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv # expected: one line per GPU, e.g. "NVIDIA GeForce RTX 4090, 24564 MiB, 550.xx" python3 --version # expected: Python 3.x, per each project's docs free -g | head -2 # expected: enough RAM for the weights you plan to load ss -ltnp | grep -E '8080|8000' # expected: empty output = both ports are free 

My checklist before continuing: nvidia-smi prints a GPU and driver version; cmake --version and g++ --version both work; python3 -m venv --help works; ports 8080 and 8000 are free; and I know the exact model file or repo ID I'm loading.

llama.cpp: build it, then serve it

git clone https://github.com/ggml-org/llama.cpp cd llama.cpp cmake -B build -DGGML_CUDA=ON # omit -DGGML_CUDA=ON for a CPU-only build cmake --build build --config Release -j 
./build/bin/llama-server \ -m ./models/model.Q4_K_M.gguf \ -c 4096 \ -ngl 99 \ --host 0.0.0.0 \ --port 8080 

-ngl 99 pushes every layer onto the GPU; set it to 0 for a pure CPU run. -c 4096 is the context window in tokens. When the server is ready it prints the address it's listening on.

vLLM: same API, different engine

python3 -m venv .venv source .venv/bin/activate pip install vllm 
vllm serve <hf-repo-id> \ --host 0.0.0.0 \ --port 8000 \ --max-model-len 4096 \ --gpu-memory-utilization 0.90 

Swap <hf-repo-id> for the repository you actually intend to serve — the model card states the ID. Add --tensor-parallel-size 2 to split the model across two GPUs. The two flags that matter most: --max-model-len caps the KV cache, and --gpu-memory-utilization is the share of VRAM vLLM is allowed to claim.

Because both engines expose the same /v1 surface, my client code stays identical. Swapping engines later means changing the base URL and the model name, nothing else.

Prove the endpoint works and write down numbers

In a second terminal, confirm the process is listening. If it isn't, scroll the server log to the last traceback — it failed during model load, not at request time.

ss -ltnp | grep -E '8080|8000' # expected: a line with LISTEN and the port you chose 

Hit /v1/models first — that only proves the server is up. Then /v1/chat/completions, which proves the weights actually loaded and generation runs.

curl -s http://localhost:8080/v1/models # expected: {"object":"list","data":[{"id":"model.Q4_K_M.gguf", ...}]} 

Copy the id from that response into the next call:

curl -s http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "model.Q4_K_M.gguf", "messages": [{"role": "user", "content": "Reply with the single word: ok"}], "max_tokens": 8 }' 

You're looking for a choices array whose message.content holds the answer, plus a usage object with token counts. Then time the round trip on a fixed prompt:

curl -s -o /dev/null -w "total=%{time_total}s\n" http://localhost:8080/v1/models 

Run that three times per engine, with the same model size and the same prompt length, and compare. I keep watch -n1 nvidia-smi open in a third terminal while generating — if memory keeps climbing until the process dies, lower --gpu-memory-utilization on vLLM or -ngl on llama.cpp.

The startup failure I keep hitting with vLLM

torch.OutOfMemoryError: CUDA out of memory. on a GPU that has enough memory on paper.

The cause is pre-allocation: vLLM reserves the KV cache according to --gpu-memory-utilization, and the default share plus the weights can exceed what's actually free — especially if another process is still holding VRAM. Lower both the share and the context, make sure nvidia-smi shows the old process gone, and restart:

vllm serve <hf-repo-id> --port 8000 --gpu-memory-utilization 0.80 --max-model-len 2048 

Takeaway

Let concurrency pick the engine. One user on one machine, or no GPU in the box: llama.cpp with a quantized GGUF. A shared endpoint with parallel callers on a server GPU: vLLM. Keep your client pointed at /v1, time both on your own hardware with the same prompt, and treat any benchmark you read online as a hint rather than a verdict.

This article was originally published on AI Release — https://ai-release.org. If you want more local-inference walkthroughs like this one, there's also a channel at https://t.me/ai_release1.