Hello!
When working with ONNX Runtime, you may run into an error like the following

[E:onnxruntime:Default, provider_bridge_ort.cc:1744 TryGetProviderInfo_CUDA] 
Failed to load library libonnxruntime_providers_cuda.so with error: 
libcublasLt.so.11: cannot open shared object file: No such file or directory

[W:onnxruntime:Default, onnxruntime_pybind_state.cc:870 CreateExecutionProviderInstance] 
Failed to create CUDAExecutionProvider.

This error indicates that GPU acceleration is unavailable and the runtime has fallen back to CPU execution. In this article, we walk through the entire process, from investigating the cause of this problem to resolving it.

Symptoms

When the error occurs

• A warning appears every time a model is loaded in ONNX Runtime
• Image processing or inference takes abnormally long (e.g., several seconds or more)
• onnxruntime-gpu is installed, yet the GPU is not being used
• The same warning appears repeatedly every time the program runs

Performance impact

# CPU execution (when the error occurs)
Model A warmup time: 5.071s
Model B warmup time: 0.029s

# Expected values for GPU execution (normal)
Model A warmup time: 0.5-1.0s  # 5-10x speedup
Model B warmup time: 0.005-0.01s  # 3-5x speedup

Investigating the Cause

Step 1: Check the current environment

First, let's confirm whether ONNX Runtime can recognize the GPU

# Check installed packages
pip list | grep onnx

Example output

onnx                     1.16.1
onnxruntime-gpu          1.18.0

# Check GPU recognition status
python -c "import onnxruntime; print(onnxruntime.get_device()); print(onnxruntime.get_available_providers())"

Example output

GPU
['TensorrtExecutionProvider', 'CUDAExecutionProvider',  'CPUExecutionProvider']

Step 2: Analyze the problem

At this point, an interesting situation emerged

• onnxruntime.get_device() → GPU (recognized)
• CUDAExecutionProvider is included in the list of available providers
• Yet an error occurs at runtime and execution falls back to the CPU

This indicates that while ONNX Runtime itself recognizes the GPU, the CUDA libraries required at runtime are missing.

Step 3: Identify the version mismatch

# Check the system's CUDA version
nvcc --version
ls -la /usr/local/ | grep cuda

What the investigation revealed

• The system already has CUDA 12 installed
• ONNX Runtime is looking for CUDA 11 libraries (libcublasLt.so.11)

So we were able to pinpoint a version mismatch as the cause.

Evaluating Solutions

Option 1: Symbolic links (assessing the risk)

Initially, we considered solving this with a symbolic link

# Make the CUDA 12 library visible as CUDA 11
sudo ln -s /lib/x86_64-linux-gnu/libcublasLt.so.12 /lib/x86_64-linux-gnu/libcublasLt.so.11

That said, is such a quick fix really acceptable? Let's consider the risks

• Advantage: quick and requires no additional installation
• Risk: potential ABI compatibility issues and impact on other applications that depend on CUDA 11

Upon further investigation, we found that
onnxruntime-gpu 1.18.0 is already a CUDA 12-compatible build.
In other words, the symbolic link stays within the same CUDA 12 family, so we judged the risk to be small

Option 2: Finding the complete solution

However, the symbolic link alone did not solve the problem, and further investigation revealed that cuDNN was missing.

With that, here is the final solution

★ The Final Solution

Step 1: Install cuDNN

conda install -c conda-forge cudnn=8.9.7.29 -y

This step is the most important: without cuDNN, CUDAExecutionProvider cannot be initialized.

Step 2: Create the symbolic links

# Create a symbolic link for libcublasLt.so.11
sudo ln -sf /usr/local/cuda-12/lib64/libcublasLt.so.12 \
            /usr/lib/x86_64-linux-gnu/libcublasLt.so.11

# Create a symbolic link for libcublas.so.11  
sudo ln -sf /usr/local/cuda-12/lib64/libcublas.so.12 \
            /usr/lib/x86_64-linux-gnu/libcublas.so.11

# Update the library cache
sudo ldconfig

Step 3: Reinstall ONNX Runtime

# Uninstall the existing packages
pip uninstall onnxruntime onnxruntime-gpu -y

# Install the GPU build
pip install onnxruntime-gpu==1.22.0

Pinning the version is recommended, since reinstalling may otherwise pull in the latest release

Why the Symbolic Links Alone Were Not Enough

Why the symbolic links alone did not solve the problem on the first attempt

1. libcublasLt.so.11 → resolved by the symbolic link
2. libcudnn.so.8 → was missing
3. Other CUDA-related libraries → incomplete

This is how we discovered that installing cuDNN was essential.

Verification

Confirm that the GPU is being used

import onnxruntime as ort

# Create a test session
session = ort.InferenceSession(
    "your_model.onnx",
    providers=['CUDAExecutionProvider', 'CPUExecutionProvider']
)

# Check which providers are actually in use
print("Active providers:", session.get_providers())
# Output on success: ['CUDAExecutionProvider', 'CPUExecutionProvider']
# Output on failure: ['CPUExecutionProvider']

Performance test

import time
import onnxruntime as ort
import numpy as np

# Load the model
session = ort.InferenceSession("model.onnx")

# Dummy input data
dummy_input = np.random.randn(1, 3, 224, 224).astype(np.float32)

# Warmup
for _ in range(5):
    session.run(None, {"input": dummy_input})

# Measure
start = time.time()
for _ in range(100):
    session.run(None, {"input": dummy_input})
print(f"Average inference time: {(time.time() - start) / 100:.4f}s")

Troubleshooting

Debugging checklist

# 1. Check the ONNX Runtime version
python -c "import onnxruntime; print(onnxruntime.__version__)"

# 2. Check GPU recognition
python -c "import onnxruntime; print(onnxruntime.get_device())"

# 3. Check available providers
python -c "import onnxruntime; print(onnxruntime.get_available_providers())"

# 4. Check the CUDA installation
nvcc --version
ls -la /usr/local/ | grep cuda

# 5. Check that the required libraries exist
ls -la /usr/lib/x86_64-linux-gnu/ | grep -E "libcublas|libcudnn"

# 6. Check dependencies
ldd $(python -c "import onnxruntime; print(onnxruntime.__file__)") | grep "not found"

Complete Environment Setup Procedure

requirements.txt

onnxruntime-gpu==1.22.0

setup.sh

#!/bin/bash

echo "=== ONNX Runtime GPU Setup ==="

# 0. Check the current state
echo "Current environment check:"
python -c "import onnxruntime; print('Version:', onnxruntime.__version__); print('Device:', onnxruntime.get_device())" 2>/dev/null || echo "ONNX Runtime not installed"

# 1. Install cuDNN
echo "Installing cuDNN..."
conda install -c conda-forge cudnn=8.9.7.29 -y

# 2. Create symbolic links
echo "Creating symbolic links..."
sudo ln -sf /usr/local/cuda-12/lib64/libcublasLt.so.12 \
            /usr/lib/x86_64-linux-gnu/libcublasLt.so.11
sudo ln -sf /usr/local/cuda-12/lib64/libcublas.so.12 \
            /usr/lib/x86_64-linux-gnu/libcublas.so.11

# 3. Update the library cache
echo "Updating library cache..."
sudo ldconfig

# 4. Install ONNX Runtime
echo "Installing ONNX Runtime GPU..."
pip uninstall onnxruntime onnxruntime-gpu -y
pip install -r requirements.txt

# 5. Verify
echo "Verification:"
python -c "
import onnxruntime as ort
print('ONNX Runtime version:', ort.__version__)
print('Device:', ort.get_device())
print('Available providers:', ort.get_available_providers())
"

echo "Setup complete!"

Summary

We reached the solution through the following steps

1. Initial diagnosis: the GPU is recognized, but an error occurs at runtime
2. Root cause: a CUDA 11/12 version mismatch plus missing cuDNN
3. Solution: install cuDNN + create symbolic links + reinstall

The key point is that the symbolic links alone were not enough — installing cuDNN was essential. By performing these three steps in the correct order, you can enable GPU acceleration and speed up inference by 2 to 10 times.
In particular, if you overlook the CPU fallback, all that GPU power goes to waste, so when this warning appears, it is well worth addressing it properly