Hello!
This is the Qualiteg Product Development Team!

Today we describe what happened when we put two RTX 4090s into a Windows + WSL2 machine and tried to run a large open model with vLLM — and got thoroughly stuck at NCCL initialization.

Information on this is scattered and fragmentary, and it took real persistence to get through, so we hope this saves some time for anyone wearing themselves out on the same configuration.

Background

Our goal this time was to evaluate the latest open models (open-weight LLMs), which keep arriving one after another, on our own hardware.

New open models appear every few weeks. Beyond the benchmark numbers, we verify hands-on how they actually behave for our use cases — output quality, speed, how much they degrade when quantized, and their strengths and weaknesses on agent-style tasks.

The environment here is a Windows + WSL2 (Ubuntu) machine fitted with two RTX 4090s (24 GB each).

nvidia-smi reports a CUDA Version of 12.8。

What we are running is a large open model quantized with AWQ(4bit).

Even at 4-bit, the weights come to tens of gigabytes and will not fit on a single card, so tensor-parallel=2 (splitting the model across the two cards) was a given from the start.

We wanted it running as a resident server we could call through an API, so we chose vLLM. It exposes an OpenAI-compatible endpoint, which means our evaluation code can call local and cloud models through the same interface.

One note:
consumer GPUs in the 40 series have no NVLink。

The two cards communicate over PCIe. You might think, "Then can't they simply talk to each other directly over PCIe (P2P)?"
That is in fact what we assumed at first. As described later, however,

we could not use PCIe-based GPU-to-GPU P2P as a valid NCCL communication path from within WSL2.

Our understanding is that this stems not from the physical wiring or the grade of the GPUs, but from the combination of WSL2's GPU virtualization layer and NCCL. This turned out to be the crux of the whole exercise.

To cut to the chase, what awaited us here were three narrow passages peculiar to WSL2.

This article walks through each in order: what we tried, what we hit, and how we got through. The second trap in particular is documented only in fragments, and it took considerable persistence to escape. If this saves someone on the same configuration a day of effort, we will consider it well worth writing.

The big picture: what is "tensor-parallel" actually doing?

First, let us clarify why "splitting across two cards" is necessary at all.

A large model's weights (parameters) alone will not fit into a single GPU's memory. The technique used here is tensor parallelism (TP).

It distributes the model's main tensor computations across multiple GPUs; within a single inference pass, each GPU computes its assigned portion, and the results are gathered and combined.

The key here is the "collective communication" that aggregates each GPU's results. For example, after computing on split matrices, an operation called all-reduce — which adds up the partial sums from all GPUs — runs frequently. The component responsible for this collective communication is NCCL (the NVIDIA Collective Communications Library). The effective performance of tensor parallelism — and indeed whether the server can start at all — hinges on NCCL initializing and communicating correctly.

The second trap described below is precisely a story of stumbling at this NCCL initialization.

When you tell vLLM at startup how many GPUs to split the model across, it internally launches one worker process per GPU, forms a communication group with NCCL, and loads each worker's share of the weights. The theory is simple — but on the WSL2 playing field, "forming a communication group with NCCL" turned out to be the gauntlet.

Trap 1: a CUDA wheel mismatch, and it will not even start

We began by installing vLLM the straightforward way.

On our machine, the driver shows up in nvidia-smi as CUDA Version 12.8. What a plain pip install pulled down, however, was a version built for the CUDA 13 series (cu130).

Because the driver is still on the CUDA 12 series, the moment you launch it you get an error to the effect that a CUDA 13 runtime shared library (such as libcudart.so.13) cannot be found, and it crashes without recognizing the GPUs at all. The real culprit was the fact that a CUDA 13 build (cu130) had been pulled in.

This is a classic trap in the machine learning world.

The lesson is simple: pin the build (wheel) that matches your driver's CUDA major version by hand, rather than leaving it to automatic dependency resolution. Since our driver is on the CUDA 12 series, we set out to align everything on CUDA 12 wheels.

A practical note is worth adding here.
From CUDA 11 onward, NVIDIA provides minor version compatibility within the same major version.

In other words, as long as the minimum driver requirement is met, a wheel bundling a different 12.x runtime can work in a CUDA 12.x driver environment. That said, this does not mean "any 12.x always works unconditionally." NVIDIA's own documentation notes it as a "limited feature-set": newer CUDA features, PTX JIT, and inter-library dependencies can still require a newer driver.

In our setup, we pinned PyTorch to cu128 and vLLM to the cu129 prebuilt wheel available for that release, specifying the wheel URLs directly. Both passed import and startup verification, so we adopted this combination.

What matters is avoiding a mismatch across major versions — such as installing a CUDA 13 (cu130) wheel into a CUDA 12 driver environment. As for minor-version differences, verifying at the end with an import and an actual startup is, in practice, good enough.

A common failure mode is skipping the version pin on the assumption that "newest must be fastest." You then receive a CUDA 13 build while your driver (CUDA 12 series) has not caught up — a mismatch. With GPUs, newer is not always better; at a minimum, avoid mismatches that cross a major version.

Verification is almost anticlimactically easy: from a Python interactive session, simply check whether the framework's native extension module imports. If the import succeeds, you are at least consistent with the CUDA runtime. Conversely, as long as this errors out, no amount of configuration will get you any further.

Locking this down first is the golden rule.

Trap 2 (the main event): NCCL fails to initialize with "unhandled cuda error"

Now we come to the heart of the matter.

When we tried to launch with the model split across the two cards, worker-process initialization failed with an error along these lines:

RuntimeError: NCCL error: unhandled cuda error

This then cascades into "failed to initialize engine core" errors, and the server never comes up.

What makes this troublesome is that the message — "unhandled cuda error" — is exactly the kind of vague error you see all the time. Reading what is actually wrong out of this message alone is close to impossible.

Start with the standard remedies

It is well known that on WSL2, GPU-to-GPU P2P (peer-to-peer communication over PCIe) and the datacenter-oriented InfiniBand family of transports often do not work out of the box.

So we first disabled the two usual suspects via environment variables.

NCCL_P2P_DISABLE=1
NCCL_IB_DISABLE=1

The hope is that NCCL will then steer around the unusable paths and fall back to alternatives such as shared memory. Most "multi-GPU on WSL" articles declare the problem solved at this point.

Except that this alone did not fix it

In this environment, however, that was not enough.

Even with both settings in place, the "unhandled cuda error" kept coming.

This is where most of the available information runs dry; frankly, isolating the cause took us close to half a day.

We enabled NCCL's verbose logging (NCCL_DEBUG=INFO) to trace which stage of communication was failing, and got as far as pinning the initialization failure down to the memory-allocation phase.

The decisive fix: disabling the cuMem allocator

What finally did the trick was this single line.

NCCL_CUMEM_ENABLE=0

This setting disables the cuMem (CUDA virtual memory management) allocator used by NCCL.

Relatively recent versions of NCCL use cuMem (CUDA Virtual Memory Management) to allocate and register memory, making communication more efficient.

In WSL2's paravirtualized GPU environment, however, this cuMem-based allocation and registration apparently fails to mesh properly, triggering errors during communication-group initialization
(NCCL_DEBUG=INFO logs confirmed the failure was occurring in the memory allocation/registration phase).

Disabling it and falling back to the conventional allocation path finally let NCCL through.

For added stability, we also put the following two measures in place.

• Pin the worker start method to spawn.
  (making the process start method explicit and avoiding fork-related initialization problems)
  
• In vLLM's launch options, disable the custom all-reduce implementation and route everything through the standard NCCL path.

So in the end,
the combination that worked is as follows.

NCCL_P2P_DISABLE=1
NCCL_IB_DISABLE=1
NCCL_CUMEM_ENABLE=0        # ← this was the decisive fix
(worker start method) spawn
(vLLM launch option) disable custom all-reduce, use the standard path

When you launch in this state, each worker loads its share of the weights in turn, followed by computation-graph compilation and CUDA graph capture to accelerate inference. This takes several minutes, so the trick is not to panic when the logs appear to stall.

If, at the end, the model-list endpoint returns your running model, you have succeeded.

Is this really "because it's WSL"?

This was the point where everything finally clicked for us.

Our conclusion is that this family of NCCL errors is
very much specific to WSL2
. The reason lies in how GPUs work under WSL2.

Under WSL2, the host Windows GPU is exposed to the Linux guest through paravirtualization (GPU-PV). On the Linux side, CUDA calls go through a virtual GPU device and are ultimately bridged to the host driver. This mechanism is a remarkable piece of engineering that makes "GPU access from Linux on Windows" possible, but compared with bare metal (Linux installed directly on the machine), the behavior of direct GPU-to-GPU communication (P2P) and low-level memory management (cuMem) can differ.

• P2P: a mechanism by which GPUs physically connected via PCIe or NVLink exchange data directly without going through the CPU. On WSL2's virtual GPUs, this direct path often cannot be established as is.
• cuMem: CUDA's virtual memory management API. Newer NCCL versions use it to allocate and register communication buffers, but it is prone to compatibility issues with a paravirtualized GPU memory space.

To sum up so far: this problem most likely cannot be avoided through GPU grade or wiring.

Our two cards are connected over PCIe, and on bare-metal Linux, PCIe-based GPU-to-GPU P2P can be enabled given the right motherboard PCIe topology and BIOS/ACS/IOMMU settings.

In the environment described in this article, however, that PCIe P2P could not be used as a valid NCCL path from the WSL2 guest. So even with NVLink-equipped GPUs, if the WSL2 side cannot see that link as a valid NCCL P2P path, you may well run into the same class of problem.

At the very least, assuming
"NVLink would definitely solve this"
or "put in two good GPUs and the communication will sort itself out" seems risky.

The problem most likely lies not in the physical layer but in the virtualization layer above it (this can depend on your environment and versions, so the reliable approach is to check the topology reported by nvidia-smi and the NCCL logs on your own configuration).

Conversely, on bare-metal Linux, neither NCCL_CUMEM_ENABLE=0 nor P2P_DISABLE is usually necessary
(with proper configuration, both PCIe P2P and cuMem generally work as intended)

. So it is fair to say that with the same model, the same GPUs, and the same card count, this is far less likely to occur on native Linux.

In other words, this whole ordeal was the price we paid for choosing the WSL2 playing field to stay comfortable on our Windows development machine.

That said, we honestly would not go so far as to claim it is 100% WSL-only — we have not tried every other environment.

In some container environments, and even on bare metal depending on IOMMU and ACS (Access Control Services) settings, P2P can end up disabled, and similar cuMem-related issues have been reported.

So the more precise framing is probably: "it tends to surface in paravirtualized or constrained environments, and WSL2 is the leading example."

WSL offers an excellent developer experience, and in exchange these low-level pitfalls occasionally surface — keeping that in mind helps you brace for them.

Trap 3: background launches vanish along with the session

Having cleared NCCL, we thought we were finally up and running — but one more hill remained.

If you launch the WSL command from the Windows side in the background (with a trailing ampersand), the moment the command invocation returns, the entire server process disappears. The process list shows nothing.

This comes down to WSL's process lifecycle.

A WSL session can be torn down when the parent process that started it exits, taking its child processes down with it.nohup (which makes the process ignore the hangup signal) does not necessarily help either — depending on how the session is torn down, the process can still be dragged down with it.

There are several directions for dealing with this.

• Keep the terminal open and hold the process in the foreground (during evaluation work, this is often all you need).
• If systemd is enabled in your WSL instance, you can also run it as a systemd service. Note, however, that the shutdown behavior of the WSL instance itself and the persistence of user services depend on your configuration, so for real use you should verify that the process survives with the session actually closed.
• Launch it from the Windows-side Task Scheduler, starting it via wsl.exe, which ties it to the Windows process tree.

If you see symptoms like "the startup log says success, but the API will not connect" or "nothing in the process list," suspecting this first will save you time. It is a mundane trap, but we suspect quite a few people lose thirty minutes here.

What we gained

Having cleared these traps, we can now load a large open model split across two RTX 4090s with tensor parallelism and keep it running as an OpenAI-compatible API.

Production serving belongs on bare metal, but being able to turn evaluation cycles around quickly on a local development machine has been a significant win for us.

Looking back, all three traps only surface when the conditions — WSL2, multi-GPU, and a quantized large model — line up, and the documentation out there covers them only in fragments.

The second one in particular — the road to NCCL_CUMEM_ENABLE=0 — was a long one, and being able to share it is, we believe, the main value of this article. "Disable P2P and IB" appears in plenty of articles; we hope this reaches those who are stuck just beyond that point.

Test environment

For reference, here are the versions used in the configuration covered in this article. NCCL issues are highly version-dependent, so if you arrived here with the same symptoms, please read this against the versions in your own environment.

PyTorch at cu128 and vLLM at cu129 being slightly out of step is not a typo. As explained in Trap 1, it is a deliberate combination — aligned on CUDA major version 12 and operated within the bounds of minor version compatibility. We confirmed it works via import and startup verification.

Summary (checklist)

As a quick reference for later, here are just the settings.

At install time

• Pin the framework build by hand to match your driver's CUDA major version. A plain pip install pulls in a CUDA 13 (cu130) build, which mismatches a CUDA 12 driver. The fix is to align on CUDA 12 (cu12x) wheels. Minor differences (such as cu128 vs. cu129) can work within CUDA minor version compatibility, but new features, PTX, and inter-library dependencies may impose additional constraints — so always confirm with an import and a startup test in the end.

Environment variables for multi-GPU startup

NCCL_P2P_DISABLE=1
NCCL_IB_DISABLE=1
NCCL_CUMEM_ENABLE=0

vLLM launch options

• Pin the worker start method to spawn.
• Disable the custom all-reduce implementation and route through the standard NCCL path.

Operational notes

• The NCCL environment variables above are workaround and triage settings, and they may cost performance. In production or on bare-metal Linux, let NCCL's automatic selection do its job first, and apply these only when needed.
• This cluster of NCCL issues appears highly specific to WSL2's paravirtualized GPU. On bare-metal Linux (with BIOS/ACS and related settings in order), they are usually unnecessary.
• Keep the server process in the foreground, or detach it from the session via systemd (user services), Task Scheduler, or similar. The latter approaches are environment-dependent, so verify that the process survives with the session actually closed. Background launches can vanish along with the session.
• After startup, loading plus compilation takes several minutes. Wait mechanically, retrying until the model-list endpoint responds.

Thank you for reading this far.
Once you are through it, a low-level pitfall amounts to a single line of configuration.
Getting to that single line, however, is a long road.

We hope this serves as at least a small shortcut for anyone wearing themselves out on the same path.

See you next time!