Daily Dev Tips

Understanding and Fixing libstdc++ Version Mismatches Between the System and conda

Qualiteg Product Development Team

Qualiteg Product Development Team

— 7 min read
Understanding and Fixing libstdc++ Version Mismatches Between the System and conda

Hello!

The other day, while building a Python application that uses dlib (running in a conda environment), I ran into the following error.

ImportError: /home/mlu/anaconda3/envs/example_env/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /home/mlu/anaconda3/envs/example_env/lib/python3.10/site-packages/_dlib_pybind11.cpython-310-x86_64-linux-gnu.so)

The error essentially says: "the dlib_pybind11 module requires GLIBCXX_3.4.32, but it cannot be found!"

In this article, we explain not only how to fix this error but also why it occurs in the first place.

First, the Fix

Let's start with the fix: the following command resolves the error.

conda install -c conda-forge libstdcxx-ng

The Mechanism Behind "GLIBCXX_x.x.x not found"

Let's uncover the mechanism by which this problem arises in real-world environment setup and package build/install scenarios

STEP 1. Start with the Anaconda installation

We installed Anaconda as follows—deliberately choosing a slightly older Anaconda.

# install anaconda
wget https://repo.anaconda.com/archive/Anaconda3-2024.02-1-Linux-x86_64.sh

bash Anaconda3-2024.02-1-Linux-x86_64.sh -b

echo "export PATH=~/anaconda3/bin:\$PATH" >> ~/.bashrc
source ~/.bashrc

With that, Anaconda is installed.

STEP 2. Install cmake, which dlib needs at install time

We will install dlib later. dlib is a Python package, but binaries get built at install time, so we install cmake on the system to make that build possible

sudo apt-get update
sudo apt install -y build-essential
sudo apt-get install -y cmake

During installation of the build-essential package, 

The following NEW packages will be installed:
  build-essential g++ g++-13 libstdc++-13-dev
  Setting up g++ (4:13.2.0-7ubuntu1) ...
update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode

and thus g++ was configured as the system's default C++ compiler

Next, here are the important excerpts from the cmake installation log

The following NEW packages will be installed:
  cmake cmake-data cpp cpp-13 cpp-13-x86-64-linux-gnu cpp-x86-64-linux-gnu gcc gcc-13 gcc-13-base
  gcc-13-x86-64-linux-gnu gcc-x86-64-linux-gnu libaom3 libarchive13t64 libasan8 libatomic1 libc-dev-bin
  libc-devtools libc6-dev libcc1-0 libcrypt-dev libde265-0 libgcc-13-dev libgd3 libgomp1
  ...
3 upgraded, 44 newly installed, 0 to remove and 165 not upgraded.
Need to get 77.4 MB of archives.
After this operation, 232 MB of additional disk space will be used.

Setting up gcc-13-x86-64-linux-gnu (13.3.0-6ubuntu2~24.04) ...
Setting up gcc-13 (13.3.0-6ubuntu2~24.04) ...
Setting up gcc-x86-64-linux-gnu (4:13.2.0-7ubuntu1) ...
Setting up gcc (4:13.2.0-7ubuntu1) ...

As this log shows, merely installing cmake caused GCC 13.3.0 to be freshly installed in its entirety

STEP 3. Create a conda virtual environment

Next, let's create a conda virtual environment to serve as the runtime for our Python application

conda create -n example_env python=3.10.0
conda init bash
source ~/.bashrc
conda activate example_env

This gives us a conda virtual environment named example_env

Now let's enter this environment and install dlib with pip

STEP 4. pip-install (and build) dlib

Installation is quite simple—just do the following

conda activate example_env
pip install dlib

Here, though, to understand exactly what is going on, let's install dlib with verbose logging enabled, like this.

conda activate example_env
# view dlib's build log in detail
pip install dlib --verbose --force-reinstall --no-cache-dir

If you see something like "CMake is not installed on your system!", you skipped STEP 2—make sure cmake is properly installed on the system

Now, when dlib builds, you will see output like the following in the log

-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Check for working C compiler: /usr/bin/cc - skipped
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Found PythonInterp: /home/mlu/anaconda3/envs/example_env/bin/python3.10
-- Found PythonLibs: /home/mlu/anaconda3/envs/example_env/lib/libpython3.10.so
/usr/include/c++/13/new:128:26: note: in a call to allocation function 'operator new []' declared here
128 | *GLIBCXX*NODISCARD void* operator new[](std::size_t) *GLIBCXX*THROW (std::bad_alloc)

So what does this tell us?
Even though pip install is running inside a conda environment

  • Compiler: uses the system's /usr/bin/cc, /usr/bin/c++ (GCC 13.3.0)
  • C++ standard library: uses the system's /usr/include/c++/13/ headers
  • Python environment: uses the Python libraries inside the conda environment

In other words, dlib

  • Build tooling (compiler) → uses the system's compiler
  • Runtime environment (Python) → uses the conda environment

That is what is happening

To sum up, the mechanism behind "GLIBCXX_x.x.x not found" is this:

a binary that depends on the system's newer GLIBCXX_3.4.32 was being run against the conda environment's older libstdc++—hence the error.

That is quite a trap, isn't it?

What really stands out is that the build happens on the system while execution happens in conda, and each ends up referencing libraries that are completely uncoordinated with the other.

What Exactly Is a conda Environment?

conda was designed to provide an "isolated runtime environment." It is not a full OS, but it carries its own set of libraries and, at runtime, prefers the libraries inside the environment. As mentioned above, conda includes its own libstdc++, but this has nothing to do with the system's copy: it is a package built independently on conda-forge, and its version is in no way coordinated with the system-side library.

What Exactly Is GLIBCXX?

It is the ABI (Application Binary Interface) version within libstdc++,
written in the form GLIBCXX_3.4.32. The number increases each time new features are added.

For example, here is the rough correspondence with GCC versions

GCC 11 → up to GLIBCXX_3.4.29
GCC 12 → up to GLIBCXX_3.4.30
GCC 13 → up to GLIBCXX_3.4.32
GCC 14 → up to GLIBCXX_3.4.33

What Exactly Is libstdcxx-ng?

libstdcxx-ng is a conda package name—it is the conda edition of the GNU C++ standard library (libstdc++)

Its version number (e.g., 13, 14, 15) corresponds to the GCC major version

How do GCC versions, libstdcxx-ng (conda's C++ standard library), and GLIBCXX_ correspond?

In summary, the versions map as follows
GCC 9 → libstdcxx-ng 9 → up to GLIBCXX_3.4.26
GCC 11 → libstdcxx-ng 11 → up to GLIBCXX_3.4.29
GCC 12 → libstdcxx-ng 12 → up to GLIBCXX_3.4.30
GCC 13 → libstdcxx-ng 13 → up to GLIBCXX_3.4.32

Verifying the Difference Between the System and conda

Now that we understand how the mismatch arises, let's verify it in practice

Examining the conda-side C++ standard library

First, let's look at the version of libstdcxx on the conda side.

$ conda list | grep -E "(gcc|libstdcxx)"

_libgcc_mutex             0.1                        main
libgcc-ng                 11.2.0               h1234567_1
libstdcxx-ng              11.2.0               h1234567_1

As you can see here, conda came with libstdcxx-ng 11.2.0 installed. That is the prebuilt C++ standard library from GCC 11.2.0.

Let's also look at the GLIBCXX versions

$ strings ~/anaconda3/lib/libstdc++.so.6 | grep GLIBCXX | tail -5

_ZNKSs15_M_check_lengthEmmPKc@@GLIBCXX_3.4.5
_ZNKSt14basic_ifstreamIwSt11char_traitsIwEE7is_openEv@GLIBCXX_3.4
_ZNSs4_Rep26_M_set_length_and_sharableEm@@GLIBCXX_3.4.5
GLIBCXX_3.4.26
_ZNKSs11_M_disjunctEPKc@GLIBCXX_3.4

There it is: GLIBCXX_3.4.26.

Now recall the error message from the beginning,

ImportError: /home/mlu/anaconda3/envs/example_env/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /home/mlu/anaconda3/envs/example_env/lib/python3.10/site-packages/_dlib_pybind11.cpython-310-x86_64-linux-gnu.so)

It says "GLIBCXX_3.4.32 not found."

Of course it does—the conda side only goes up to GLIBCXX_3.4.26.

That is our culprit.

Examining the system-side C++ standard library

Now let's find out which C++ standard library the dlib in question (that is, /home/mlu/anaconda3/envs/example_env/lib/python3.10/site-packages/_dlib_pybind11.cpython-310-x86_64-linux-gnu.so) is linked against

# Here, $CONDA_PREFIX is /home/mlu/anaconda3/envs/example_env
$ldd $CONDA_PREFIX/lib/python3.10/site-packages/_dlib_pybind11.cpython-310-x86_64-linux-gnu.so | grep libstdc++

        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f9085cab000)

/lib/x86_64-linux-gnu/libstdc++.so.6That is the output—sure enough, dlib was linked against the system's libstdc++, and it is now clear that it is not linked against the conda environment's $CONDA_PREFIX/lib/libstdc++.so.6.

And when we check the GLIBCXX versions on the system side, sure enough, GLIBCXX_3.4.32 is there.

strings /lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBCXX_3.4.32

GLIBCXX_3.4.32

(You can also run strings /lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBCXX to list the GLIBCXX versions in the system's libstdc++)

Once More: The Fix

We showed the fix for this problem at the beginning of the article

conda install -c conda-forge libstdcxx-ng

This brings the C++ standard library inside conda up to the latest version. However, given that

GCC 12 → libstdcxx-ng 12 → up to GLIBCXX_3.4.30
GCC 13 → libstdcxx-ng 13 → up to GLIBCXX_3.4.32

the GCC 13 standard library is really all conda needs here. So rather than chasing the very latest version, it is enough to ensure GCC 13 gets installed, as shown below

conda install -c conda-forge libstdcxx-ng=13

Summary

In this article we walked through a real-world case of dependency hell with the Python package dlib—"the build environment and the runtime environment use different library versions"—covering its cause and its remedy.
Because we installed a slightly old Anaconda, the C++ standard library bundled with it was old as well, while the system carried a newer standard library. The build ran against the system (in fact, that happens automatically), and as a result the binary would not run in the conda environment.

Many Python packages involve building a C++ layer with Linux and GCC, so dependency problems like this arise frequently. Dependencies are troublesome enough in pure Python, and when system-layer build dependencies (here, the libstdc++ library) act up, the instinct is to groan "what a pain." But if you understand how these problems arise and how building and linking actually work, you can troubleshoot with your feet on solid ground. Leave this area fuzzy, and you risk sinking into even deeper dependency hell—we hope this article helps you resolve exactly these kinds of headaches!

See you next time!

Read more