Ollama Docker Troubleshooting: Fix GPU, OOM, Networking, and Volume Issues

Your Ollama container won’t see the GPU. It dies mid-inference with no warning. Or you redeploy and every model you pulled is just gone. Sound familiar? You’re not doing anything unusual. These four failure modes account for almost every Ollama-in-Docker support thread out there. Each one has a distinct fingerprint and a specific fix. This runbook walks through all of them.

This is a troubleshooting runbook, not an install guide. It assumes Docker is already running and an Ollama container is up (or trying to come up) on a Linux host. Notes for Windows and macOS follow where the behavior differs.

Prerequisites

Before working through the fixes below, confirm you have:

Docker Engine 27.x or later (docker --version) with Docker Compose v2
An existing Ollama container, image ollama/ollama:latest or pinned tag
SSH or terminal access to the Docker host with sudo privileges
For GPU troubleshooting: an NVIDIA GPU with driver 550+ installed on the host, or an AMD GPU with ROCm 7.x
Basic familiarity with docker logs, docker exec, and docker volume commands

Test environment used for the examples below: Ubuntu 24.04 LTS, Docker 27.3.1, NVIDIA driver 550.120, nvidia-container-toolkit 1.17.x.

Quick Diagnosis

Before chasing a specific fix, run this five-step check. It takes under two minutes and rules out three of the four failure modes right away.

# 1. Check the container logs for explicit errors
docker logs ollama --tail 50

# 2. Confirm Docker can see the GPU at all (independent of Ollama)
docker run --rm --gpus all ubuntu nvidia-smi

# 3. Check host-level memory pressure and OOM kills
dmesg | tail -30

# 4. Test API reachability from inside the container's network
docker exec -it ollama curl -s http://localhost:11434

# 5. Confirm your models are on a named volume, not the container's writable layer
docker volume ls

Match what you see against this table:

What you observe	Likely failure mode	Jump to
`nvidia-smi` fails or isn’t found in step 2	GPU passthrough is broken	GPU Not Detected
GPU worked before, logs now show CPU fallback	Cgroup driver conflict	GPU Falls Back to CPU
`dmesg` shows `Out of memory: Killed process`	Memory exhaustion	OOM Crashes
curl in step 4 hangs or refuses from another container	Networking/binding issue	Connectivity Issues
`docker volume ls` shows no Ollama volume	Models stored in writable layer	Model Persistence

Terminal showing the five-step diagnostic commands run in sequence with their output, used to triage which Ollama Docker failure mode is occurring

Common Issues

GPU Not Detected in the Ollama Container

Symptoms:

ollama run responds, but noticeably slower than expected
docker logs ollama shows no mention of CUDA or GPU, or explicitly says no GPU detected, falling back to CPU
Host nvidia-smi works fine outside any container

Why it happens: Docker doesn’t pass GPUs to containers automatically. Three things need to line up: the NVIDIA driver on the host, the nvidia-container-toolkit installed and registered with Docker, and the --gpus all flag on the container itself. Miss any one of those, and Ollama won’t throw an error. It just quietly falls back to CPU inference. That’s exactly why this issue shows up in so many support threads.

Fix:

First, figure out if this is a Docker problem or an Ollama problem. Test GPU passthrough with a plain Ubuntu image:

docker run --rm --gpus all ubuntu nvidia-smi

Expected output on a working setup:

+—————————————————————————–+
| NVIDIA-SMI 550.120 Driver Version: 550.120 CUDA Version: 12.4 |
|——————————-+———————-+———————-+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| 0 NVIDIA RTX 4070 Off | 00000000:01:00.0 On | N/A |
+——————————-+———————-+———————-+

If instead you get docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]], the nvidia-container-toolkit isn’t installed or registered. Install it:

# Add the NVIDIA container toolkit repo (Ubuntu/Debian)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit

# Register the toolkit with the Docker daemon and restart it
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Re-run the nvidia-smi test above. Once it passes, recreate your Ollama container with --gpus all included:

docker run -d --name ollama \
  --gpus all \
  -e OLLAMA_HOST=0.0.0.0 \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  ollama/ollama:latest

Terminal output of docker run --gpus all ubuntu nvidia-smi succeeding, showing GPU name and driver version

If nvidia-smi works inside the container but Ollama still reports CPU-only, the NVIDIA UVM kernel module is probably not loaded correctly on the host:

sudo nvidia-modprobe -u
sudo rmmod nvidia_uvm
sudo modprobe nvidia_uvm
sudo reboot

Verification:

docker exec -it ollama nvidia-smi

You should see the same GPU listing as the host-level test. Then check Ollama’s own logs for the discovery line:

docker logs ollama 2>&1 | grep -i "gpu\|cuda"

Expected: a line referencing your GPU model and available VRAM, not no compatible GPUs were discovered.

Tip: If you’re on AMD hardware instead of NVIDIA, the fix is different. Pass --device /dev/kfd --device /dev/dri and match --group-add values for the render and video groups. You’ll also need a current ROCm 7.x driver on the host. Verify with docker exec -it ollama rocm-smi instead of nvidia-smi.

GPU Falls Back to CPU After Initially Working

Symptoms:

Inference was fast right after container creation
Performance degrades over hours or days, or after a host reboot
docker logs ollama shows GPU detection at startup but CPU-only inference on later requests

Why it happens: This one catches people off guard, because the container did have working GPU access at first. The usual culprit is a mismatch between Docker’s cgroup driver and systemd’s cgroup management. When they conflict, GPU device cgroup rules can silently drop after a daemon restart, a systemd unit reload, or a kernel update. Ollama doesn’t throw an error. It just stops seeing the device.

Fix:

Check your current cgroup driver:

docker info | grep -i cgroup

If it shows Cgroup Driver: systemd and you’re seeing intermittent GPU loss, switch Docker to cgroupfs to remove the conflict:

// /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=cgroupfs"]
}

sudo systemctl restart docker
docker start ollama

Verification:

docker logs ollama --tail 20

Look for a clean GPU discovery message with no fallback warning. Run an inference request and time it. A 7B Q4 model should return the first token in well under a second on a mid-range GPU. Multi-second latency means you’re still on CPU.

Terminal output of docker logs ollama showing GPU detection line versus a CPU fallback warning line, for comparison

OOM Crashes or Freezing Under Load

Symptoms:

Container exits unexpectedly with code 137
docker inspect ollama --format '{{.State.OOMKilled}}' returns true
Requests hang indefinitely when multiple models or concurrent requests are in flight

Why it happens: Ollama happily loads multiple models into memory and runs requests in parallel, as long as it thinks it has enough RAM or VRAM. On a shared homelab box, or a GPU with limited VRAM (8–12 GB is common), that assumption breaks fast. Two environment variables control this: OLLAMA_NUM_PARALLEL (concurrent request slots per model) and OLLAMA_MAX_LOADED_MODELS (how many models stay loaded at once). OLLAMA_NUM_PARALLEL defaults to 1, and OLLAMA_MAX_LOADED_MODELS defaults to three times the number of GPUs detected (or 3 on CPU-only setups). Even with those conservative defaults, a container with no hard memory limit set can still get squeezed once you raise OLLAMA_NUM_PARALLEL or load several models at once.

Fix:

Confirm it’s actually an OOM kill and not a crash from another cause:

docker inspect ollama --format '{{.State.OOMKilled}}'

true

Also check the host kernel log for the kill event:

dmesg | tail -30

[12345.678] Out of memory: Killed process 48213 (ollama) total-vm:18234012kB

Recreate the container with tighter concurrency limits:

docker rm -f ollama

docker run -d --name ollama \
  --gpus all \
  -e OLLAMA_HOST=0.0.0.0 \
  -e OLLAMA_NUM_PARALLEL=1 \
  -e OLLAMA_MAX_LOADED_MODELS=1 \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --memory=12g \
  ollama/ollama:latest

Comment on the flags: OLLAMA_NUM_PARALLEL=1 processes one request at a time per model instead of batching several. OLLAMA_MAX_LOADED_MODELS=1 forces Ollama to unload one model before loading another. The --memory=12g flag caps the container’s RAM usage. That way, a runaway process gets killed cleanly by Docker instead of taking down other services on the host.

If you’re still tight on memory, switch to a more heavily quantized model. A Q4_K_M build of a 7B model needs roughly 4.5 GB of RAM or VRAM, versus 14+ GB for an f16 build. That difference is often the gap between an OOM crash and a stable deployment, at the small cost of precision.

ollama pull llama3.1:8b-instruct-q4_K_M

Verification:

docker stats ollama --no-stream

CONTAINER CPU % MEM USAGE / LIMIT MEM %
ollama 2.10% 3.2GiB / 12GiB 26.67%

Run a few concurrent requests and confirm memory stays under the limit instead of climbing until the container dies.

Tip: Environment variables are only read at container startup. Changing them requires a docker rm and recreate; a docker restart will not apply them.

Other Containers or Apps Can’t Reach the Ollama API

Symptoms:

curl to Ollama works fine from the Docker host itself
The same request from another container, or from a device elsewhere on your network, times out or gets connection refused
A web UI like Open WebUI shows “cannot connect to Ollama server”

Why it happens: By default, Ollama binds to 127.0.0.1, which only accepts connections from inside the same network namespace. Inside a container, that means only processes in that same container can reach it. Not other containers, not the host, and not other machines on your LAN. You need to tell it explicitly to listen on all interfaces.

Fix:

Set OLLAMA_HOST=0.0.0.0 on the container so it binds to every interface instead of just loopback:

docker run -d --name ollama \
  --gpus all \
  -e OLLAMA_HOST=0.0.0.0 \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  ollama/ollama:latest

How the connecting app should reach it depends on where it’s running:

Another container on the same Docker network/Compose stack: use the service name, e.g. http://ollama:11434. This is the cleanest option, with no host networking quirks involved.
Another container not on the same network, or a script run directly on the host: on Linux, host.docker.internal doesn’t work by default the way it does on Docker Desktop. Add it explicitly:

docker run -d --name open-webui \
  --add-host=host.docker.internal:host-gateway \
  -p 8080:8080 \
  ghcr.io/open-webui/open-webui:main

Then point the app at http://host.docker.internal:11434.

A device elsewhere on your LAN: use the Docker host’s actual IP address, e.g. http://192.168.1.50:11434, and make sure the host firewall allows inbound connections on port 11434.

Verification:

docker exec -it open-webui curl -s http://host.docker.internal:11434

Ollama is running

Terminal output of a curl request from a second container to the Ollama API returning the "Ollama is running" response, confirming connectivity

host.docker.internal vs. the host IP, in short: host.docker.internal is a DNS alias Docker resolves to the host’s internal gateway address. It’s convenient because it doesn’t change even if your host’s LAN IP does. On Linux, though, it needs the --add-host flag; Docker Desktop wires it in automatically, but plain Docker on Linux doesn’t. The host’s real IP always works without extra flags, but it breaks the moment DHCP hands out a new address. For anything long-lived, use a static IP or DHCP reservation, or just stick to Docker service names on a shared network.

Warning: Binding to 0.0.0.0 exposes the API to anything that can reach that port. If the host is on a shared or untrusted network, put Ollama behind a reverse proxy with authentication, or restrict access with host firewall rules (ufw allow from 192.168.1.0/24 to any port 11434) rather than exposing it to the open network.

Downloaded Models Disappear After Recreating the Container

Symptoms:

You ran ollama pull llama3.1 and it worked
After docker rm and recreating the container (or pulling a new image tag), the model list is empty
Re-pulling seems to “fix” it, but the problem repeats on every update

Why it happens: Ollama stores pulled models under /root/.ollama inside the container. Skip mounting a volume there, and that data lives in the container’s writable layer. Docker deletes that layer permanently the moment you remove the container. That’s just how Docker works. Containers are disposable by design, and only volumes survive removal.

Fix:

Check whether you’re currently using a named volume:

docker volume ls

DRIVER VOLUME NAME
local ollama

If you don’t see an ollama volume (or whatever you named it), your models are unprotected. Recreate the container with a named volume mounted at the data directory:

docker rm -f ollama

docker run -d --name ollama \
  --gpus all \
  -e OLLAMA_HOST=0.0.0.0 \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  ollama/ollama:latest

The -v ollama:/root/.ollama flag creates (or reuses) a named volume called ollama and mounts it at the path Ollama uses to store models and configuration. Docker manages named volumes directly, separate from any container’s lifecycle. You can docker rm the container as many times as you want. The volume stays put.

Re-pull your models once, and they’ll survive from here on:

ollama pull llama3.1:8b-instruct-q4_K_M

Verification:

docker volume inspect ollama

[
    {
        "Name": "ollama",
        "Driver": "local",
        "Mountpoint": "/var/lib/docker/volumes/ollama/_data",
        "Scope": "local"
    }
]

Terminal output of docker volume ls and docker volume inspect ollama showing the named volume and its Mountpoint field

Test it directly, recreate the container again and confirm the model list is intact:

docker rm -f ollama
docker run -d --name ollama --gpus all -e OLLAMA_HOST=0.0.0.0 -v ollama:/root/.ollama -p 11434:11434 ollama/ollama:latest
docker exec -it ollama ollama list

NAME ID SIZE MODIFIED
llama3.1:8b-instruct-q4_K_M a1b2c3d4e5f6 4.9 GB 2 minutes ago

Example docker run command visible in terminal with --gpus all, -e OLLAMA_HOST=0.0.0.0, and -v ollama:/root/.ollama flags highlighted

A Specific Model Fails to Load or Errors Out After Pulling

Symptoms:

ollama run <model> fails with a checksum or manifest error
The model appeared in ollama list but inference immediately errors out

Why it happens: A download got interrupted somewhere, such as a network blip, a container restart mid-pull, or disk space running out, and left a partially written model file behind.

Fix:

ollama rm llama3.1:8b-instruct-q4_K_M
ollama pull llama3.1:8b-instruct-q4_K_M

Verification:

ollama run llama3.1:8b-instruct-q4_K_M "Say hello in five words."

You should get a normal text response instead of a manifest or checksum error.

Error Messages Table

Error Message	Likely Cause	Fix
`could not select device driver "" with capabilities: [[gpu]]`	nvidia-container-toolkit not installed/registered	Install toolkit, run `nvidia-ctk runtime configure --runtime=docker`, restart Docker
`no compatible GPUs were discovered` (in `docker logs`)	Container started without `--gpus all`, or UVM module not loaded	Add `--gpus all` to `docker run`; reload `nvidia_uvm` module
Container exit code `137`	Out-of-memory kill	Lower `OLLAMA_NUM_PARALLEL` / `OLLAMA_MAX_LOADED_MODELS`, add `--memory` limit, use a quantized model
`connection refused` from another container	Ollama bound to `127.0.0.1` instead of `0.0.0.0`	Set `-e OLLAMA_HOST=0.0.0.0` and recreate the container
`curl: (7) Failed to connect to host.docker.internal` (Linux)	Missing host gateway alias	Add `--add-host=host.docker.internal:host-gateway` to the connecting container
Empty `ollama list` after container recreation	Models stored in writable layer, no volume mounted	Recreate with `-v ollama:/root/.ollama`, re-pull models
`manifest unknown` or checksum mismatch on run	Interrupted or corrupted download	`ollama rm <model>` then `ollama pull <model>` again

Platform-Specific Issues

Linux

Linux is the primary target for GPU-accelerated Ollama containers, and most of the fixes above assume it. One Linux-specific gotcha: host.docker.internal does not resolve automatically the way it does on Docker Desktop. You always need --add-host=host.docker.internal:host-gateway on the connecting container, or use the host’s real interface IP instead.

# Find the host IP visible from inside the Docker bridge network
ip -4 addr show docker0

docker0: <BROADCAST,MULTICAST,UP,LOWER_UP>
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0

Terminal output of ip addr show docker0 revealing the Docker bridge network host IP used for container-to-host connectivity

Windows

On Windows Server or Windows 11 with Docker Desktop, GPU passthrough for NVIDIA cards runs through WSL2’s GPU paravirtualization, not nvidia-container-toolkit directly on the host. Confirm the WSL2 backend is active in Docker Desktop settings. You’ll also need a current Game Ready or Studio driver, not the older WSL-specific driver package that Microsoft deprecated. host.docker.internal resolves automatically here. You don’t need the --add-host workaround that Linux requires. If GPU still isn’t detected, run wsl --update from PowerShell and restart Docker Desktop.

wsl --update
wsl --shutdown

macOS

Docker Desktop on macOS runs containers inside a Linux VM. That VM has no path to Apple Silicon or discrete GPUs. --gpus all does nothing here, and there’s no workaround for it. If you need GPU acceleration on a Mac, run Ollama natively, outside Docker, using Apple’s Metal backend instead. Networking and volume troubleshooting steps here still apply unchanged. They’re part of Docker Desktop’s networking layer, not host-specific behavior. host.docker.internal works out of the box on macOS, same as Windows.

Configuration Issues

A few misconfigurations show up over and over. They’re worth calling out on their own.

Binding to loopback instead of all interfaces:

# Wrong — only reachable from inside the same container
-e OLLAMA_HOST=127.0.0.1

# Correct — reachable from other containers and hosts
-e OLLAMA_HOST=0.0.0.0

Mounting the wrong path for persistence:

# Wrong — models still get lost, wrong internal path
-v ollama-data:/data/ollama

# Correct — matches Ollama's actual data directory inside the container
-v ollama:/root/.ollama

Setting environment variables on a running container and expecting them to apply:

# This does nothing — env vars are read once at startup
docker exec ollama env OLLAMA_NUM_PARALLEL=1

# Correct — recreate the container with the variable set
docker rm -f ollama
docker run -d --name ollama -e OLLAMA_NUM_PARALLEL=1 -v ollama:/root/.ollama -p 11434:11434 ollama/ollama:latest

Getting Help

If you’ve worked through the sections above and you’re still stuck, gather this before asking for help:

# Full container logs
docker logs ollama > ollama-logs.txt

# Container configuration and resource limits
docker inspect ollama > ollama-inspect.json

# Enable verbose debug logging for a fresh reproduction
docker rm -f ollama
docker run -d --name ollama -e OLLAMA_DEBUG=1 -v ollama:/root/.ollama -p 11434:11434 ollama/ollama:latest
docker logs -f ollama

Reproduce the issue with OLLAMA_DEBUG=1 set. Then check the logs for the specific error rather than a generic timeout. The debug output usually names the exact GPU device, memory allocation, or network bind that failed.

Worth bookmarking: the Ollama Troubleshooting Guide and the Ollama GitHub repository issue tracker. The tracker is the fastest place to check if you’re hitting a known bug tied to a specific Ollama version.

Prevention Tips

Always mount a named volume from the start: Add -v ollama:/root/.ollama the first time you create the container, not after you’ve already lost a model library.
Pin your Ollama image tag: Using ollama/ollama:latest means a background update can change GPU or memory behavior without warning. Pin a specific version tag and upgrade deliberately.
Set memory limits proactively: Don’t wait for an OOM kill to add --memory and tune OLLAMA_NUM_PARALLEL/OLLAMA_MAX_LOADED_MODELS. Set conservative values up front, especially on shared hosts.
Test GPU passthrough independent of Ollama: Keep the docker run --gpus all ubuntu nvidia-smi command handy as a first check after any host driver or Docker update.
Document your networking topology: If you’re running Ollama alongside Open WebUI or other clients, put them on the same Docker Compose network with defined service names instead of relying on host.docker.internal, which adds a layer of host-networking dependency you don’t need.

Wrapping Up

Most Ollama-in-Docker problems come down to one of four things: broken GPU passthrough, no memory limits set, the API bound to the wrong interface, or models that were never persisted to a volume. Once you know which bucket you’re in, the fix is usually a one-line flag change and a container recreate.

Run the five-step diagnostic at the top of this article before you start guessing. It’ll tell you which bucket you’re in faster than reading through all five sections looking for your symptoms.

Step	Action	Applies To
1	Run `docker run --gpus all ubuntu nvidia-smi`	GPU not detected
2	Fix cgroup driver mismatch in `daemon.json`	GPU falls back to CPU
3	Set `OLLAMA_NUM_PARALLEL` / `OLLAMA_MAX_LOADED_MODELS`, use quantized models	OOM crashes
4	Set `OLLAMA_HOST=0.0.0.0`, use correct hostname for connecting apps	Connectivity issues
5	Mount `-v ollama:/root/.ollama` before pulling models	Model persistence

Last updated: 2026-09-11 | Applies to Ollama running in Docker on Linux (Ubuntu/Debian), Windows Server/Docker Desktop, and macOS Docker Desktop