CUDA Initialization Error: cudaGetDeviceCount returned 802 on Ubuntu 24.04 with NVIDIA Driver 560.x and CUDA 12.6 [closed]

I'm encountering an issue when trying to run CUDA applications on a Dell PowerEdge XE9680 with NVIDIA GPUs. The system is running Ubuntu 24.04, and I've installed NVIDIA driver version 560.x along with CUDA 12.6. However, when I attempt to run the deviceQuery sample from the CUDA toolkit, I receive the following error:

cudaGetDeviceCount returned 802-> system not yet initializedResult = FAIL

System Configuration:

• Server Model: Dell PowerEdge (H100 GPU)
• Operating System: Ubuntu 24.04
• NVIDIA Driver Version: 560.x
• CUDA Version: 12.6

Steps I’ve Tried:

1. Verified that the NVIDIA driver is correctly installed by running nvidia-smi, which correctly displays the GPUs and driver version.
2. Reinstalled both the NVIDIA drivers and CUDA toolkit to ensure no conflicts or installation issues.
3. Checked for compatibility between the installed CUDA version and the NVIDIA driver.
4. Reset the NVIDIA driver using systemctl and manually reloaded the kernel modules.
5. Rebuilt the initramfs and rebooted the system.
6. Verified that the nvidia kernel modules are loaded correctly using lsmod | grep nvidia.

Despite these efforts, I continue to receive the same error when running CUDA applications. The error suggests that the system has not yet initialized, but I’m unable to determine the root cause.

Questions:

1. Has anyone encountered a similar issue, and what steps did you take to resolve it?
2. Are there specific logs or diagnostics that could provide more insight into why CUDA is not initializing properly?
3. Could this issue be related to a specific configuration on the Dell PowerEdge XE9680, or is it more likely related to the software environment?

Any guidance or suggestions would be greatly appreciated. I'm open to trying additional troubleshooting steps or providing more information if needed.