--- title: "Ollama Docker Setup: GPU, Volumes and Compose" published: true description: "Ollama Docker setup tested on Docker 29.4.3 and Ollama 0.40.13: persistent volumes, GPU flags for NVIDIA and AMD, a Compose file and the gotchas." tags: ollama, docker, devops, ai canonical_url: https://devtoolhub.com/ollama-docker-setup-gpu-volumes/ --- > Originally published on [DevToolHub](https://devtoolhub.com/ollama-docker-setup-gpu-volumes/). Running Ollama in a container takes one `docker run` command. That command also decides two things the docs bury: whether your models survive a rebuild, and whether your GPU gets used. This Ollama Docker setup guide covers both. I ran it on Docker 29.4.3 with Ollama 0.40.13 and hit three problems the one-liner doesn't warn you about. ## What Is the Minimum Ollama Docker Setup? The minimum Ollama Docker setup is one container with a named volume and port 11434 published. The official [Ollama Docker docs](https://docs.ollama.com/docker) give this CPU-only command, and it worked as written: 

 First, check that the API answers, then pull and run a model inside the container: 

 The first curl returns `Ollama is running`. The server logs show what hardware it found, and on my Mac the line read `inference compute id=cpu`. That log line is your first GPU check. The second check is `docker exec ollama ollama ps`, whose `PROCESSOR` column read `100% CPU` for a loaded model. Inside the container, Ollama binds to `0.0.0.0:11434` by default. That is why the published port works without setting `OLLAMA_HOST` yourself. ## How Do Volumes Keep Ollama Models Between Containers? A named volume mounted at `/root/.ollama` keeps your models when you delete and recreate the container. Without it, every `docker rm` throws away every model you pulled, and the next container starts empty. I tested both cases with a 270 MB model, `smollm2:135m`: 

 However, the volume holds more than models. Mounting a bind directory showed four entries in it: `models`, `cache`, `id_ed25519` and `id_ed25519.pub`. Ollama generates that key pair on first start ("Couldn't find '/root/.ollama/id_ed25519'. Generating new private key" in the log). So if you lose the volume, you lose the key too, not just the downloads. ⚠️ **Note:** the volume name changes between `docker run` and Compose. Compose prefixes volume names with the project name, so a volume called `ollama` in `compose.yaml` became `comp_ollama` on my machine. Models you pulled with the plain `docker run` command above won't show up. Set `name: ollama` under the volume to pin it, as in the file further down. ## How Do You Enable GPU in an Ollama Docker Setup? For NVIDIA, install the NVIDIA Container Toolkit on the host and register it with Docker. Then add `--gpus=all` to the run command. AMD uses the `rocm` image tag with the `/dev/kfd` and `/dev/dri` devices. Docker Desktop on macOS has no GPU passthrough at all. ### NVIDIA Ollama Docker Setup Ollama's docs walk through the toolkit install for apt and yum, and the [NVIDIA install guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) has the current repository steps. After installing the package, point Docker at it and restart the daemon: 

 Then start the container with every GPU visible: 

 On top of that, your card needs compute capability 5.0 or higher and a driver version 550 or newer, according to the [Ollama GPU docs](https://docs.ollama.com/gpu). Cards on compute capability 5.0 through 6.2 need driver 570 or newer. ### AMD Ollama Docker Setup 

 Plus, the `rocm` tag is the smaller download. Docker Hub lists the `0.40.13` amd64 image at 3.39 GB and `0.40.13-rocm` at 1.45 GB. ### Ollama Docker Setup on macOS Ollama's FAQ is blunt: GPU acceleration is not available for Docker Desktop on macOS. On my Mac, `--gpus=all` failed with `failed to discover GPU vendor from CDI: no known GPU vendor found`. So if you want Apple GPU speed, run Ollama natively and keep Docker for everything else. [Ollama vs LM Studio](https://devtoolhub.com/ollama-vs-lm-studio/) covers the native install. ### How do you confirm the container is really using the GPU? Run a prompt, then check `docker exec ollama ollama ps`. A `PROCESSOR` value of `100% GPU` means the model sits entirely in VRAM. `100% CPU` means no GPU was found. A split like `48%/52% CPU/GPU` means the model sits partly in each. Because of that, a healthy status proves nothing about the GPU. ## How Do You Write an Ollama Docker Compose File? An Ollama Docker Compose file needs four things beyond the image. Those are a named volume, a localhost port binding, a healthcheck, and an optional one-shot service that pulls your model. This is the file I tested, with the GPU block split out into its own file so the base file runs anywhere: 

 Start it on a CPU host with `docker compose up -d`. On a GPU host, layer the second file on top: 

 ### Why use `ollama list` as the healthcheck in an Ollama Docker setup? The image ships without `curl` or `wget`, so the usual `curl -f http://localhost:11434` healthcheck fails with a missing binary. `ollama list` talks to the local server, so it works as a probe. With it, `pull-model` waited for the `Healthy` state, pulled the model, and exited with code 0. ### Why does the GPU block get its own file? Because Compose refuses to start the service on a host without the NVIDIA runtime. With the `deploy` block in the single file, `docker compose up` on my Mac stopped with `could not select device driver "nvidia" with capabilities: [[gpu]]`. The Docker docs say the `capabilities` field is required. They also say `count` and `device_ids` are mutually exclusive. Check those two first if your GPU block fails on a real GPU host. See the [Compose GPU support page](https://docs.docker.com/compose/how-tos/gpu-support/) for the full field list. ### Why bind the port to 127.0.0.1? Because `-p 11434:11434` publishes on every interface. After a plain run, `docker port` showed `0.0.0.0:11434` and `[::]:11434`. The `127.0.0.1:11434:11434` form in the file above showed `127.0.0.1:11434` only. Docker also publishes ports ahead of `ufw` rules, which its own docs cover under [Docker and ufw](https://docs.docker.com/engine/network/packet-filtering-firewalls/). If this host faces the internet, read [the Ollama VPS hardening guide](https://devtoolhub.com/ollama-production-vps-hardening/) before you publish anything. ## Which Environment Variables Matter in Docker? Five variables cover most tuning, and you set them with `-e` on `docker run` or under `environment:` in Compose. Defaults below come from the server config line Ollama prints at startup and from the [Ollama FAQ](https://docs.ollama.com/faq). | Variable | Default | What it does | |---|---|---| | `OLLAMA_KEEP_ALIVE` | `5m` | How long an idle model stays in memory | | `OLLAMA_NUM_PARALLEL` | `1` | Parallel requests per model; RAM scales with this times the context length | | `OLLAMA_MAX_LOADED_MODELS` | 3 per GPU, or 3 on CPU | Models loaded at once, if they fit | | `OLLAMA_MODELS` | `/root/.ollama/models` | Where models live inside the container | | `OLLAMA_VULKAN` | `true` | Vulkan GPU support, bundled in the image | One more detail from the startup log: with no GPU, Ollama picked a default context of 4096 tokens (`default_num_ctx=4096`). Set `OLLAMA_CONTEXT_LENGTH` if your prompts need more, and size memory for it. [Ollama hardware requirements](https://devtoolhub.com/ollama-hardware-requirements/) has the RAM and VRAM math. ## What Breaks in an Ollama Docker Setup? Three things broke or surprised me, and the first one costs the most. ⚠️ **`docker compose down -v` deletes the models volume, and the `-v` flag is the only difference.** Plain `docker compose down` stopped the containers and left the `ollama` volume in place. Adding `-v` removed it. Models and key pair went with it. Use it only when you want a clean slate: 

 The other two: - **`latest` moves fast.** Docker Hub showed `0.40.12` on 27 October, release candidates on the 28th, and `0.40.13` on the 29th. Pin the tag in Compose so a restart doesn't pull a new version while you're debugging. - **But the image is big.** The amd64 image is 3.39 GB compressed on Docker Hub. The arm64 image used 4.29 GB on my disk once pulled. Budget for that before the first model. ## What I Did Not Test I tested everything on a Mac with no NVIDIA, AMD or Vulkan GPU. So none of the GPU paths ran against real hardware. I did not run the NVIDIA toolkit install, the `rocm` image, Vulkan, Windows with WSL2 or Jetson. Multi-GPU selection with `device_ids` is untested too. Those steps come from the official docs I linked, and the only GPU behavior I observed was the failure messages above. Everything else in this guide ran as written. ## Frequently Asked Questions **Q: How do I run Ollama in Docker with GPU support?** A: On NVIDIA, install the NVIDIA Container Toolkit, run `sudo nvidia-ctk runtime configure --runtime=docker`, restart Docker, then start the container with `--gpus=all`. On AMD, use the `ollama/ollama:rocm` image with `--device /dev/kfd --device /dev/dri`. Confirm with `ollama ps`. **Q: Where does Ollama store models in Docker?** A: Inside the container at `/root/.ollama/models`. Mount a named volume at `/root/.ollama` so models survive container removal. The same directory holds the `id_ed25519` key pair Ollama generates on first start. **Q: Why is Ollama using the CPU inside Docker?** A: Ollama found no GPU the container can access. Check that you passed `--gpus=all` or the AMD devices. Check that the host driver meets the version requirement. Then run `docker exec ollama ollama ps` and look for `100% CPU`. Docker Desktop on macOS can never pass a GPU through. **Q: Can I use Ollama in Docker on a Mac?** A: Yes, on the CPU only. Ollama's FAQ states GPU acceleration is unavailable for Docker Desktop on macOS. Run Ollama natively for Apple GPU performance. **Quick Summary:** - Mount a named volume at `/root/.ollama`, or every container removal deletes your models and the key pair. - Compose prefixes volume names, so set `name: ollama` to share one volume between `docker run` and Compose. - Check the GPU with `docker exec ollama ollama ps`, not the container's health status. - Put the NVIDIA `deploy` block in a separate override file, because the single-file version fails on hosts without the runtime. - The image has no `curl`, so use `ollama list` as the healthcheck, and bind the port to `127.0.0.1` unless you mean to expose it. Your Ollama Docker setup is now running, so point your apps at it with the [Ollama API guide](https://devtoolhub.com/ollama-api-guide/).