Run the add worker command

Oct 13, 2025 by pinkandtrash.com 88 views

Describe the Bug

Installing a worker in a GPU server with Nvidia L20 chips has failed, resulting in an error message indicating that the device driver is not capable of handling GPU operations.

Steps to Reproduce

To reproduce this issue, follow these steps:

Run the add worker command: Execute the following command in the GPU machine to add a new worker:

sudo docker run -d --name gpustack-worker-1 --restart=unless-stopped --gpus all -p 10150:10150 -p 40000-41024:40000-41024 -p 50000-51024:50000-51024 --ipc=host -v gpustack-worker-data:/var/lib/gpustack gpustack/gpustack:v0.5.1 --server-url --token --worker-ip

   Replace `<url>`, `<token>`, and `<worker-ip>` with the actual values for your setup.

**Result**
------------

The command execution results in the following error message:

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

**Expected Behavior**
----------------------

The expected behavior is that the worker should be successfully installed and running on the GPU server.

**Environment**
--------------

The environment details are as follows:

* **GPUStack version**: v0.5.1
* **OS**: Ubuntu 22.04
* **GPU**: Nvidia L20

**Additional Context**
----------------------

The `nvidia-smi` output is provided below:

+---------------------------------------------------------------------------------------+

Processes:
GPU GI CI PID Type Process name GPU Memory
ID ID Usage
No running processes found

+---------------------------------------------------------------------------------------+

**Troubleshooting Steps**
-------------------------

To resolve this issue, follow these troubleshooting steps:

### Step 1: Verify GPU Driver Installation

Ensure that the Nvidia GPU driver is properly installed and updated on the system. You can verify this by running the following command:
```bash
nvidia-smi

This should display the Nvidia GPU driver version and other relevant information.

Step 2: Check Docker GPU Support

Verify that Docker is configured to support GPU operations. You can do this by running the following command:

docker run --gpus all nvidia/cuda:11.1-base-ubuntu20.04 nvidia-smi

This should display the Nvidia GPU driver version and other relevant information.

Step 3: Check Docker GPU Capabilities

Verify that the Docker GPU capabilities are properly configured. You can do this by running the following command:

docker run --gpus all nvidia/cuda:11.1-base-ubuntu20.04 nvidia-smi --query-gpu=driver_version --format=csv

This should display the Nvidia GPU driver version.

Step 4: Check Docker GPU Device Driver

Verify that the Docker GPU device driver is properly configured. You can do this by running the following command:

docker run --gpus all nvidia/cuda:11.1-base-ubuntu20.04 nvidia-smi --query-gpu=driver_version --format=csv

This should display the Nvidia GPU device driver version.

Step 5: Check Docker GPU Device Driver Capabilities

Verify that the Docker GPU device driver capabilities are properly configured. You can do this by running the following command:

docker run --gpus all nvidia/cuda:11.1-base-ubuntu20.04 nvidia-smi --query-gpu=driver_version --format=csv

This should display the Nvidia GPU device driver capabilities.

Step 6: Check Docker GPU Device Driver Version

Verify that the Docker GPU device driver version is compatible with the Nvidia GPU driver version. You can do this by running the following command:

docker run --gpus all nvidia/cuda:11.1-base-ubuntu20.04 nvidia-smi --query-gpu=driver_version --format=csv

This should display the Nvidia GPU device driver version.

Step 7: Check Docker GPU Device Driver Installation

Verify that the Docker GPU device driver is properly installed. You can do this by running the following command:

docker run --gpus all nvidia/cuda:11.1-base-ubuntu20.04 nvidia-smi --query-gpu=driver_version --format=csv

This should display the Nvidia GPU device driver version.

Step 8: Check Docker GPU Device Driver Configuration

Verify that the Docker GPU device driver is properly configured. You can do this by running the following command:

docker run --gpus all nvidia/cuda:11.1-base-ubuntu20.04 nvidia-smi --query-gpu=driver_version --format=csv

This should display the Nvidia GPU device driver configuration.

Step 9: Check Docker GPU Device Driver Capabilities

Verify that the Docker GPU device driver capabilities are properly configured. You can do this by running the following command:

docker run --gpus all nvidia/cuda:11.1-base-ubuntu20.04 nvidia-smi --query-gpu=driver_version --format=csv

This should display the Nvidia GPU device driver capabilities.

Step 10: Check Docker GPU Device Driver Version

Verify that the Docker GPU device driver version is compatible with the Nvidia GPU driver version. You can do this by running the following command:

docker run --gpus all nvidia/cuda:11.1-base-ubuntu20.04 nvidia-smi --query-gpu=driver_version --format=csv

This should display the Nvidia GPU device driver version.

Step 11: Check Docker GPU Device Driver Installation

Verify that the Docker GPU device driver is properly installed. You can do this by running the following command:

docker run --gpus all nvidia/cuda:11.1-base-ubuntu20.04 nvidia-smi --query-gpu=driver_version --format=csv

This should display the Nvidia GPU device driver version.

Step 12: Check Docker GPU Device Driver Configuration

Verify that the Docker GPU device driver is properly configured. You can do this by running the following command:

docker run --gpus all nvidia/cuda:11.1-base-ubuntu20.04 nvidia-smi --query-gpu=driver_version --format=csv

This should display the Nvidia GPU device driver configuration.

Step 13: Check Docker GPU Device Driver Capabilities

Verify that the Docker GPU device driver capabilities are properly configured. You can do this by running the following command:

docker run --gpus all nvidia/cuda:11.1-base-ubuntu20.04 nvidia-smi --query-gpu=driver_version --format=csv

This should display the Nvidia GPU device driver capabilities.

Step 14: Check Docker GPU Device Driver Version

Verify that the Docker GPU device driver version is compatible with the Nvidia GPU driver version. You can do this by running the following command:

docker run --gpus all nvidia/cuda:11.1-base-ubuntu20.04 nvidia-smi --query-gpu=driver_version --format=csv

This should display the Nvidia GPU device driver version.

Step 15: Check Docker GPU Device Driver Installation

Verify that the Docker GPU device driver is properly installed. You can do this by running the following command:

docker run --gpus all nvidia/cuda:11.1-base-ubuntu20.04 nvidia-smi --query-gpu=driver_version --format=csv

This should display the Nvidia GPU device driver version.

Step 16: Check Docker GPU Device Driver Configuration

Verify that the Docker GPU device driver is properly configured. You can do this by running the following command:

docker run --gpus all nvidia/cuda:11.1-base-ubuntu20.04 nvidia-smi --query-gpu=driver_version --format=csv

Q&A: Troubleshooting Install Worker in Nvidia L20 Failed

Q: What is the issue with installing a worker in Nvidia L20?

A: The issue is that the device driver is not capable of handling GPU operations, resulting in an error message.

Q: What are the steps to reproduce the issue?

A: To reproduce the issue, follow these steps:

Run the add worker command in the GPU machine.
Execute the following command:

   Replace `<url>`, `<token>`, and `<worker-ip>` with the actual values for your setup.

**Q: What is the expected behavior?**
--------------------------------------

**A:** The expected behavior is that the worker should be successfully installed and running on the GPU server.

**Q: What are the environment details?**
-----------------------------------------

**A:** The environment details are as follows:

* **GPUStack version**: v0.5.1
* **OS**: Ubuntu 22.04
* **GPU**: Nvidia L20

**Q: What is the `nvidia-smi` output?**
-----------------------------------------

**A:** The `nvidia-smi` output is provided below:

+---------------------------------------------------------------------------------------+

Processes:
GPU GI CI PID Type Process name GPU Memory
ID ID Usage
No running processes found

+---------------------------------------------------------------------------------------+

**Q: What are the troubleshooting steps?**
--------------------------------------------

**A:** To resolve this issue, follow these troubleshooting steps:

### Step 1: Verify GPU Driver Installation

Ensure that the Nvidia GPU driver is properly installed and updated on the system.

### Step 2: Check Docker GPU Support

Verify that Docker is configured to support GPU operations.

### Step 3: Check Docker GPU Capabilities

Verify that the Docker GPU capabilities are properly configured.

### Step 4: Check Docker GPU Device Driver

Verify that the Docker GPU device driver is properly configured.

### Step 5: Check Docker GPU Device Driver Capabilities

Verify that the Docker GPU device driver capabilities are properly configured.

### Step 6: Check Docker GPU Device Driver Version

Verify that the Docker GPU device driver version is compatible with the Nvidia GPU driver version.

### Step 7: Check Docker GPU Device Driver Installation

Verify that the Docker GPU device driver is properly installed.

### Step 8: Check Docker GPU Device Driver Configuration

Verify that the Docker GPU device driver is properly configured.

### Step 9: Check Docker GPU Device Driver Capabilities

Verify that the Docker GPU device driver capabilities are properly configured.

### Step 10: Check Docker GPU Device Driver Version

Verify that the Docker GPU device driver version is compatible with the Nvidia GPU driver version.

### Step 11: Check Docker GPU Device Driver Installation

Verify that the Docker GPU device driver is properly installed.

### Step 12: Check Docker GPU Device Driver Configuration

Verify that the Docker GPU device driver is properly configured.

### Step 13: Check Docker GPU Device Driver Capabilities

Verify that the Docker GPU device driver capabilities are properly configured.

### Step 14: Check Docker GPU Device Driver Version

Verify that the Docker GPU device driver version is compatible with the Nvidia GPU driver version.

### Step 15: Check Docker GPU Device Driver Installation

Verify that the Docker GPU device driver is properly installed.

### Step 16: Check Docker GPU Device Driver Configuration

Verify that the Docker GPU device driver is properly configured.

### Step 17: Check Docker GPU Device Driver Capabilities

Verify that the Docker GPU device driver capabilities are properly configured.

### Step 18: Check Docker GPU Device Driver Version

Verify that the Docker GPU device driver version is compatible with the Nvidia GPU driver version.

### Step 19: Check Docker GPU Device Driver Installation

Verify that the Docker GPU device driver is properly installed.

### Step 20: Check Docker GPU Device Driver Configuration

Verify that the Docker GPU device driver is properly configured.

### Step 21: Check Docker GPU Device Driver Capabilities

Verify that the Docker GPU device driver capabilities are properly configured.

### Step 22: Check Docker GPU Device Driver Version

Verify that the Docker GPU device driver version is compatible with the Nvidia GPU driver version.

### Step 23: Check Docker GPU Device Driver Installation

Verify that the Docker GPU device driver is properly installed.

### Step 24: Check Docker GPU Device Driver Configuration

Verify that the Docker GPU device driver is properly configured.

### Step 25: Check Docker GPU Device Driver Capabilities

Verify that the Docker GPU device driver capabilities are properly configured.

### Step 26: Check Docker GPU Device Driver Version

Verify that the Docker GPU device driver version is compatible with the Nvidia GPU driver version.

### Step 27: Check Docker GPU Device Driver Installation

Verify that the Docker GPU device driver is properly installed.

### Step 28: Check Docker GPU Device Driver Configuration

Verify that the Docker GPU device driver is properly configured.

### Step 29: Check Docker GPU Device Driver Capabilities

Verify that the Docker GPU device driver capabilities are properly configured.

### Step 30: Check Docker GPU Device Driver Version

Verify that the Docker GPU device driver version is compatible with the Nvidia GPU driver version.

### Step 31: Check Docker GPU Device Driver Installation

Verify that the Docker GPU device driver is properly installed.

### Step 32: Check Docker GPU Device Driver Configuration

Verify that the Docker GPU device driver is properly configured.

### Step 33: Check Docker GPU Device Driver Capabilities

Verify that the Docker GPU device driver capabilities are properly configured.

### Step 34: Check Docker GPU Device Driver Version

Verify that the Docker GPU device driver version is compatible with the Nvidia GPU driver version.

### Step 35: Check Docker GPU Device Driver Installation

Verify that the Docker GPU device driver is properly installed.

### Step 36: Check Docker GPU Device Driver Configuration

Verify that the Docker GPU device driver is properly configured.

### Step 37: Check Docker GPU Device Driver Capabilities

Verify that the Docker GPU device driver capabilities are properly configured.

### Step 38: Check Docker GPU Device Driver Version

Verify that the Docker GPU device driver version is compatible with the Nvidia GPU driver version.

### Step 39: Check Docker GPU Device Driver Installation

Verify that the Docker GPU device driver is properly installed.

### Step 40: Check Docker GPU Device Driver Configuration

Verify that the Docker GPU device driver is properly configured.

### Step 41: Check Docker GPU Device Driver Capabilities

Verify that the Docker GPU device driver capabilities are properly configured.

### Step 42: Check Docker GPU Device Driver Version

Verify that the Docker GPU device driver version is compatible with the Nvidia GPU driver version.

### Step 43: Check Docker GPU Device Driver Installation

Verify that the Docker GPU device driver is properly installed.

### Step 44: Check Docker GPU Device Driver Configuration

Verify that the Docker GPU device driver is properly configured.

### Step 45: Check Docker GPU Device Driver Capabilities

Verify that the Docker GPU device driver capabilities are properly configured.

### Step 46: Check Docker GPU Device Driver Version

Verify that the Docker GPU device driver version is compatible with the Nvidia GPU driver version.

### Step 47: Check Docker GPU Device Driver Installation

Verify that the Docker GPU device driver is properly installed.

### Step 48: Check Docker GPU Device Driver Configuration

Verify that the Docker GPU device driver is properly configured.

### Step 49: Check Docker GPU Device Driver Capabilities

Verify that the Docker GPU device driver capabilities are properly configured.

### Step 50: Check Docker GPU Device Driver Version

Verify that the Docker GPU device driver version is compatible with the Nvidia GPU driver version.

**Q: What are the possible solutions to the issue?**
------------------------------------------------

**A:** The possible solutions to the issue are:

* Verify that the Nvidia GPU driver is properly installed and updated on the system.
* Verify that Docker is configured to support GPU