Native FP8 and FP4 Support, and "fp8" Quantization with vLLM

Hello, this is the Product Development Department at Qualiteg.

When a new model is released, we often go hunting for the right recipe to speed up inference — quantizing with various techniques and switching between multiple inference engines. In the course of that work, we ran into the following issue.

Since we do not keep every GPU under the sun on hand either, it was one of those "ah, so that's what's going on" moments — so we decided to write it up in this post.

The error we encountered

When trying to load a certain LLM with vLLM 0.5.1, the following error occurred:

ValueError: The quantization method fp8 is not supported for the current GPU. Minimum capability: 89. Current capability: 86 

The cause: a GPU generation that does not support FP8

The GPU is an NVIDIA RTX A6000, and the error occurs when you try to start the OpenAI-compatible server with "fp8" quantization specified, as shown below.

python3 -m vllm.entrypoints.openai.api_server --model cyberagent/calm3-22b-chat --max-num-seqs 12 --quantization fp8 --chat-template="~/jinja/calm3_22b_chat.jinja"

The cause was that the RTX A6000 does not natively support FP8.

In other words, vLLM's FP8 quantization option only works when the hardware supports FP8 arithmetic.

(Incidentally, when the hardware does not support it, there is an escape hatch called fp8_marlin, but that would complicate the story, so we will cover it in a separate article.)

In other words, the GPU we used this time, the A6000, has a capability level of 86 (capability list), so the punch line is that it simply did not support FP8 quantization.

GPUs with native support for FP8 arithmetic

FP8 (8-bit floating-point arithmetic) is a precision added starting with Hopper, so it is available on GPUs such as the following.

What native FP4 support will bring

Furthermore, Blackwell — capability 100 — adds native FP4 support, so vLLM will presumably add native FP4 support as well.

When that happens, these latest GPUs will offer "native" quantization as an alternative to "classical" quantization methods like AWQ and GPTQ, which do not assume dedicated hardware acceleration. How large the difference turns out to be will be very interesting to see.