Setting Up Docker with Nvidia GPU

Aug 8, 2025 · 2 min read · ollama docker gotchas gpu ·

If you are trying to use Open WebUI with ollama and probably already have an nvidia-gpu and after executing following command

1docker run -d --net=host --gpus all -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:cuda

You encounter following exception

could not select device driver "" with capabilities: [[gpu]]. .

Solution

Execute these commands in the order given

1curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg   && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list |     sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' |     sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

1sudo apt-get update && sudo apt install -y nvidia-docker2

1sudo systemctl daemon-reload && sudo systemctl restart docker

And finally execute nvidia-smi

1docker run --rm  --gpus all ubuntu nvidia-smi

Terminal should pring something like this

 1+-----------------------------------------------------------------------------------------+
 2| NVIDIA-SMI xxxxxxxxxx             Driver Version: xxxxxxxxxx     CUDA Version: 12.4     |
 3|-----------------------------------------+------------------------+----------------------+
 4| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
 5| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
 6|                                         |                        |               MIG M. |
 7|=========================================+========================+======================|
 8|   0  NVIDIA GeForce RTX xxxx        Off |   00000000:xx:00.0  On |                  N/A |
 9|  0%   52C    P8             xxW /  xxxW |     xxxMiB /  xxxxxMiB |      5%      Default |
10|                                         |                        |                  N/A |
11+-----------------------------------------+------------------------+----------------------+
12                                                                                         
13+-----------------------------------------------------------------------------------------+
14| Processes:                                                                              |
15|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
16|        ID   ID                                                               Usage      |
17|=========================================================================================|
18+-----------------------------------------------------------------------------------------+

Have fun developing with