Skip to content
Friendly disclaimer: flozi00 TechHub is a solo side-project next to a full-time job — personal learning notes, no official statements. Verify critical steps yourself.

NVIDIA GPU Containers on Ubuntu 24.04

Install a host driver, Docker Engine and the NVIDIA Container Toolkit using their current official instructions, then verify GPU access.

1 min readflozi00
linuxubuntunvidiadockergpuguide
Skip to the full article
Watch · explore · understand

Three layers connect a container to the GPU

Check the host driver, the container engine and GPU access separately before testing the real CUDA application.

AI-assisted learning visualization · schematic article explanation

Step 1 / 4Make the host GPU work first

Install the driver appropriate for the hardware and verify it after reboot. If host nvidia-smi fails, diagnose that layer before adding Docker.

SystemCurrent stepSelect a block to explain it
Host NVIDIA driver
The host kernel driver must initialize successfully.
Physical NVIDIA GPU
Host nvidia-smi checks that the driver sees the device.
  1. Host NVIDIA driver Physical NVIDIA GPU
    operates device
What to remember

Driver health, Docker health and CUDA application compatibility are separate checks along one access path.

Rootless Docker uses different configuration steps. A visible GPU table does not prove every CUDA image works, and NVSwitch systems may need matching Fabric Manager components.

Terms and the complete walkthrough
Host driver
Software on the host that controls the physical GPU.
Container runtime
The software path that starts a container with its devices and resources.
Rootless Docker
Docker running its daemon and containers without root privileges.
  1. Make the host GPU work first

    Install the driver appropriate for the hardware and verify it after reboot. If host nvidia-smi fails, diagnose that layer before adding Docker.

  2. Verify Docker, then configure integration

    Run a basic container to check Docker independently. Install the NVIDIA Container Toolkit and follow the configuration path for the actual rootful or rootless daemon.

  3. Expose the host’s GPU access

    The toolkit makes requested GPU devices and driver libraries available inside the container. A successful container nvidia-smi check shows this access path works.

  4. Test the real application next

    The CUDA application still needs a compatible image and driver. A full host CUDA development toolkit is only a separate requirement when you need to build CUDA programs on the host.

All connections

  • Host NVIDIA driver → Physical NVIDIA GPU: operates device
  • Host NVIDIA driver → NVIDIA Container Toolkit: host driver access
  • Docker Engine → NVIDIA Container Toolkit: runtime configuration
  • NVIDIA Container Toolkit → GPU-enabled container: devices and libraries
  • GPU-enabled container → Actual CUDA application: application compatibility
Set your own pace. Pause at any time or select a step directly.

This guide covers Ubuntu 24.04 with an NVIDIA GPU and Docker Engine. Three components have different jobs: the host driver operates the GPU, Docker Engine runs containers, and the NVIDIA Container Toolkit makes the driver and GPU devices available inside them. A full CUDA development toolkit on the host is not required merely to run CUDA-enabled containers; install it separately only if you compile CUDA programs on the host.

1. Install and check the host driver

Follow Ubuntu's NVIDIA driver installation guide, which distinguishes general desktop use from compute/server use and recommends the appropriate package for the detected hardware. Do not assume one fixed driver branch or the open-kernel module works for every NVIDIA GPU. After installing and rebooting, run:

bash
nvidia-smi

If that fails, fix the host driver first. Docker cannot compensate for a driver that does not initialize. On NVSwitch systems, also consult NVIDIA's Fabric Manager guide for matching components and service diagnostics.

2. Install Docker Engine

Use Docker's current Ubuntu installation instructions, including its official apt repository, package names and conflict checks. The repository definition and available package versions can change; follow the maintained instructions rather than copying an old third-party repository recipe. Verify Docker independently:

bash
sudo docker run --rm hello-world

3. Install and configure the NVIDIA Container Toolkit

Follow NVIDIA's Container Toolkit installation guide to add its signed repository and install nvidia-container-toolkit. For a regular, rootful Docker daemon, NVIDIA documents:

bash
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

The first command updates Docker's daemon configuration. Rootless Docker has different configuration steps; do not use the rootful commands for that setup.

Finally, run the sample workload from NVIDIA's documentation:

bash
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

If this prints the GPU table, the container can access the host's NVIDIA driver. It does not prove that every CUDA application or image is compatible with that driver; check the application's CUDA requirements separately.

For a CUDA initialization failure on a multi-GPU system, see the diagnostic guide.