Daily Dev Tips

Verifying TensorRT-LLM v0.11.0.dev2024051400

Qualiteg Product Development Team

Qualiteg Product Development Team

— 2 min read
Verifying TensorRT-LLM v0.11.0.dev2024051400
Photo by Timur Garifov / Unsplash

Hello from the Product Development Team at Qualiteg.

TensorRT-LLM is an inference engine from NVIDIA that can be regarded as the successor to FasterTransformer, and it is available as one of the inference engines for our ChatStream.

Like vLLM, it adds support for new models quickly and already supports a wide range of existing models.

A large commit landed yesterday, so we ran a check to confirm it works. (It adds support for the multimodal models Neva and Kosmos-2, among other things.)

The model architectures supported by TensorRT-LLM are as follows.

LLM

Baichuan, BART, BERT, Blip2, BLOOM, ChatGLM, DBRX, FairSeq NMT, Falcon, Flan-T5, Gemma, GPT, GPT-J, GPT-Nemo, GPT-NeoX, InternLM, LLaMA, LLaMA-v2, Mamba, mBART, Mistral, MPT, mT5, OPT, Phi-1.5/Phi-2, Qwen, Qwen-VL, Replit Code, RoBERTa, SantaCoder, Skywork, Smaug, StarCoder, T5, Whisper

Multimodal

BLIP2 w/ OPT-2.7B, BLIP2 w/ T5-XL, CogVLM, Deplot, Fuyu, Kosmos-2, LLaVA-v1.5-7B, NeVA, Nougat family Nougat-small, Nougat-base, VILA

Verification

To provide a stable inference environment, we continuously verify the latest TensorRT-LLM builds. This time, too, we verified the latest version using a dedicated Docker container.

Today we started with a manual check, which we'll walk through here.

Start the TensorRT-LLM container.
This assumes the model files and so on are placed under /home/mlu/TensorRT-LLM on the Ubuntu host.

docker run --rm -it --ipc=host --ulimit memlock=-1 --ulimit stack=67108864  \
                --gpus=all \
                --volume /home/mlu/TensorRT-LLM:/code/tensorrt_llm \
                --env "CCACHE_DIR=/code/tensorrt_llm/cpp/.ccache" \
                --env "CCACHE_BASEDIR=/code/tensorrt_llm" \
                --workdir /app/tensorrt_llm \
                --hostname LLM-Inf-Dev-release \
                --name tensorrt_llm-release-mlu \
                --tmpfs /tmp:exec \
                tensorrt_llm_qs_ready

Move to the llama2-chat sample directory familiar from the TensorRT-LLM quick start.

cd /code/tensorrt_llm/examples/llama/

Run inference.
Let's ask for recommended spots in Asakusa.

python3 ../run.py --engine_dir ./llama-2-7b-engine  \
--max_output_len 1024 \
--tokenizer_dir ./meta-llama/Llama-2-7b-chat-hf \
--input_text "What are the recommended tourist spots in Asakusa?"

The results are shown in the video below.

Qualiteg's ChatStream supports Classic Transformer, vLLM, DeepSpeed, and TensorRT-LLM as inference engines.

You can choose the optimal inference engine depending on your requirements for high-speed LLM serving, low GPU memory usage, distributed inference, and quantization.

If you have questions or concerns about LLM inference environments or serving, please feel free to get in touch.

Read more