Hello!

In this post, we look at how to resolve the "libcudnn.so.9: cannot open shared object file" error that can occur during GPU inference with ONNX Runtime.

When running GPU inference with ONNX Runtime, you may run into a CUDA provider initialization error. This article explains what causes the error and how to fix it.

The Error Message in Detail

[E:onnxruntime:Default, provider_bridge_ort.cc:2195 TryGetProviderInfo_CUDA] 
/onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1778 
onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : 
Failed to load library libonnxruntime_providers_cuda.so with error: 
libcudnn.so.9: cannot open shared object file: No such file or directory

What Causes the Error

This error occurs in the following situations

1. cuDNN 9 is not installed: cuDNN 9 (libcudnn.so.9), which ONNX Runtime requires when running on CUDA 12, is not present on the system
2. Library path issues: cuDNN is installed, but ONNX Runtime cannot find it

This usually appears in the logs as a mere warning, but if left unaddressed, GPU inference will not run — you end up with either a CPU fallback or outright failures.

A common scenario:if you ignore the log messages, the workload silently falls back to the CPU without you ever noticing, and processing becomes strangely slow.

"Hmm, why is this so slow?"

— and it turns out the job had quietly been running on the CPU all along. This happens more often than you might think.

How to Diagnose the Problem

1. Checking cuDNN at the System Level

First, let's check for cuDNN at the system level

# Check whether it is registered as a shared library
ldconfig -p | grep libcudnn

# Check whether the physical files exist
ls -l /usr/lib/x86_64-linux-gnu/libcudnn.so* 2>/dev/null

If these commands produce no output, cuDNN is either not present on the system or not on the library path

2. Checking with the Package Manager

Next, let's check with the package manager

For APT (Ubuntu/Debian)

dpkg -l | grep libcudnn

For Conda environments

conda activate your_environment
conda list cudnn

Again, if nothing is printed, cuDNN is not installed.

How to Fix It

Method 1: Fixing It in a Conda Environment (Recommended)

If you are using a Conda environment,
you can resolve the issue by installing cuDNN inside the environment.

# 1. Activate your Conda environment
conda activate your_environment

# 2. Install cuDNN 9.10.1.4
conda install -c conda-forge cudnn=9.10.1.4 -y

Main packages that will be installed

• cuda-nvrtc-12.9.86
• cudnn-9.10.1.4
• libcublas-12.9.1.4
• libcudnn-9.10.1.4
• libcudnn-dev-9.10.1.4

Method 2: Fixing It at the System Level

Alternatively, you can install cuDNN system-wide (on Linux or WSL environments)

# For Ubuntu/Debian
sudo apt update
sudo apt install libcudnn9 libcudnn9-dev

# Refresh the library path
sudo ldconfig

Important: Reinstall ONNX Runtime

Once cuDNN has been installed via Method 1 or Method 2, reinstall onnxruntime-gpu.

After installing cuDNN, reinstall ONNX Runtime so that it correctly recognizes the new environment

# Uninstall the existing packages
pip uninstall onnxruntime onnxruntime-gpu -y

# Reinstall the GPU version
pip install onnxruntime-gpu

This reinstallation allows ONNX Runtime to correctly detect and link against the newly installed cuDNN libraries.

Verifying the Fix

You can confirm that the problem is resolved with the following Python script

import onnxruntime as ort

# List the available providers
providers = ort.get_available_providers()
print("Available providers:", providers)

# Check for the CUDA provider
if 'CUDAExecutionProvider' in providers:
    print("GPU inference is available")
    
    # Verify session creation with a simple test model
    import numpy as np
    from onnxruntime import InferenceSession
    
    try:
        # Create a session (replace with your actual model path)
        session_options = ort.SessionOptions()
        providers_list = ['CUDAExecutionProvider', 'CPUExecutionProvider']
        print("CUDAExecutionProvider initialized successfully")
    except Exception as e:
        print(f"Initialization error: {e}")
else:
    print("CUDA provider is not available")

Troubleshooting

If the Problem Persists

The steps above resolve most cases, but if the problem persists, try the following

1. Check version compatibility
   CUDA and cuDNN versions map to each other as shown below. Start by confirming version compatibility
   • CUDA 12.x → cuDNN 9.x
   • CUDA 11.x → cuDNN 8.x
   • Confirm that your ONNX Runtime GPU version is compatible with your CUDA version

Getting detailed debug information

import onnxruntime as ort
ort.set_default_logger_severity(0)  # Enable verbose logging

Setting environment variables
Setting environment variables sometimes resolves the issue

export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH

Checking the CUDA version

nvidia-smi
nvcc --version

Summary

ONNX Runtime's CUDA provider error is caused primarily by a missing cuDNN library.

If you are using a Conda environment, install cuDNN inside the environment,
and then, to be on the safe side, reinstall ONNX Runtime afterward.

In most cases, this should resolve the issue