LLM Inference Infrastructure Provisioning Course, Part 4: Selecting an Inference Engine
Hello! In the previous installments of this course, we covered in detail how to estimate the request volume an LLM service needs to handle and how to calculate the memory a model consumes during inference. This time we dig into the fourth of the seven steps: selecting an inference engine.
LLM Inference Infrastructure Provisioning Course: Series Index
- Part 1: Basic Concepts and Inference Speed
- Part 2: Estimating Request Volume for an LLM Service
- Part 3: Estimating Inference-Time Memory Consumption for Your Model
- Part 4: Selecting an Inference Engine
- Part 5: A Practical Process from GPU Node Configuration to Load Testing
- Bonus: KV Cache Offloading Strategies and GQA
What Is an Inference Engine?
An inference engine is a specialized software program designed to run LLM inference computation (text generation) efficiently on GPUs. Inference is certainly possible with general-purpose deep learning frameworks (PyTorch, TensorFlow, and the like), but in production environments a dedicated inference engine can deliver substantial performance gains and better resource efficiency.
An inference engine is more than just a runtime: it implements a variety of optimization techniques. Some engines specialize in optimizations for particular model architectures, some focus on maximizing inference speed, and others provide the KV cache memory-efficiency features we discussed last time. Each has a distinct character, so it is important to select an engine that matches the LLM model you have adopted, your operating environment, and your requirements.
An Approach to Selecting an Inference Engine
Ideally, you would select an inference engine by actually trying multiple candidates and benchmarking them. When time or resources are limited, however, starting with a rough narrowing-down pass is also an effective approach.
Compatibility with the LLM model you have adopted is especially important. By eliminating engines that do not support your planned LLM at the outset, you can narrow the field for subsequent load testing. The pairing of model and engine matters a great deal—not every inference engine delivers optimal performance with every model.
Once compatibility is confirmed, evaluate the candidates from perspectives such as performance (inference speed, memory efficiency), ease of use (setup difficulty, quality of documentation), scalability (support for distributed inference), and support (community activity, availability of commercial support).
Comparing the Major Inference Engines
Many inference engines are available on the market. Here we take a detailed look at the major engines supported by ChatStream, our LLM inference system. (You can of course use each of these engines on its own, without ChatStream.)
What Is ChatStream?
ChatStream is our one-stop solution for building LLM services, designed for constructing multi-user, large-scale LLM services like ChatGPT and Claude. It covers everything from the UI to the backend inference environment and load balancing, letting you efficiently develop not only chat applications but any kind of LLM service using local LLMs.
Each inference engine comes with its own strengths and weaknesses.
1. QCT (Qualiteg Classic Transformer)
Overview
An inference engine developed by Qualiteg, built on HF Transformers (Hugging Face Transformers). While it can be characterized as a classic engine with a conventional inference architecture, it is designed with an emphasis on broad model compatibility.
Advantages
- It runs stably with almost every Transformer-based LLM, making it a good fit when model compatibility is the priority.
For example, it runs stably even with LLMs that many other engines do not support (or support only partially, with frequent glitches).
When FUGAKU-LLM was released, for instance, we made it available in our environment, where it was tried out by many users, including government agencies. - It can run even on GPUs more than a decade old, letting you leverage existing hardware assets.
- Setup is relatively simple, requiring little special environment configuration.
Disadvantages
- Because it does not employ the latest optimization techniques, inference is slower than with modern engines.
- The number of requests it can process simultaneously (concurrency) is limited, so it is not well suited to handling heavy request volumes.
- Its memory-efficiency features are limited, which poses challenges for large models and long-context processing.
2. vLLM
Overview
A high-speed inference engine featuring an innovative KV cache management technique called PagedAttention. Developed by a research team at UC Berkeley in 2023 and released as open source, it gained wide adoption in a short time and is now one of the most popular inference engines.
Advantages
- It delivers fast inference, achieving 2–5x speedups over conventional engines.
- The PagedAttention technique greatly improves KV cache memory efficiency, increasing the number of requests that can be processed concurrently.
- Development remains highly active, with new features and performance improvements shipping frequently.
Disadvantages
- It has the constraint that a model will not run unless its attention head dimension (dim_head) is among the supported values.
- It assumes a Linux environment; deploying it on Windows requires extra workarounds.
- Older GPUs are not supported—a relatively recent NVIDIA GPU is required.
3. DeepSpeed
Overview
A high-speed inference engine developed by Microsoft, with strengths in distributed training and inference for large models. A rival of sorts to vLLM, it is particularly well suited to running in large-scale distributed environments.
Advantages
- It delivers fast inference performance and excels especially at distributed inference across multiple GPUs.
- Proprietary memory optimization techniques such as ZeRO-Inference make it possible to run large models even with limited GPU memory.
- Continued development and support from Microsoft can be expected.
Disadvantages
- It tends to support fewer model types than vLLM.
- Setup and configuration are complex, which can be a high hurdle for beginners.
- Support for the newest models can lag somewhat.
4. Text Generation Inference (TGI)
Overview
An inference engine developed by Hugging Face, tightly integrated with the company's Transformers library. It supports many models and is a particularly good match if you are already invested in the Hugging Face ecosystem.
Advantages
- It provides fast inference performance and is especially efficient for short-to-medium context lengths.
- It supports a large number of models published on the Hugging Face Hub, which is convenient when experimenting with various LLMs.
- It is designed to integrate easily with other Hugging Face tools and services.
Disadvantages
- Setup is complex, and initial configuration in particular can take time.
- With models that have not been optimized for it, inference speed can fall behind other engines.
- For large-scale distributed deployments, its feature set can trail vLLM and DeepSpeed.
5. TensorRT-LLM
Overview
A high-speed inference engine developed by NVIDIA and optimized for its own GPUs. It is designed to extract maximum performance from the latest NVIDIA GPUs such as the A100 and H100.
Advantages
- Optimized to squeeze every last drop of performance out of NVIDIA GPUs, it achieves ultra-fast inference that surpasses vLLM and DeepSpeed, reaching best-in-class speeds especially on the latest NVIDIA GPUs.
- It implements a rich set of parallelization techniques, making efficient use of multiple GPUs.
- Continued optimization and updates from NVIDIA can be expected.
Disadvantages
- It is NVIDIA-GPU-only, so you are locked into NVIDIA hardware.
- Setup is very complex and often requires specialized expertise.
- Being exclusive to NVIDIA GPUs, it cannot be used with hardware from other GPU vendors (AMD, Intel, and so on).
- Models cannot be used as distributed; a conversion process is required to turn them into engine-specific models, and not every model converts smoothly. Think of it as "compiling" a model to optimize it for NVIDIA GPUs.
(Well-known open models are already packaged as NVIDIA NIM microservices, so if you want to use open models as-is, riding on NVIDIA's cloud platform is an alternative to wrangling TensorRT-LLM yourself.)
A Practical Approach to Engine Selection
In real projects, we recommend selecting an inference engine through the following steps
- Clarify requirements
Define the requirements of your LLM service (inference speed, concurrent request capacity, supported models, scalability, and so on). - Confirm compatibility
List the inference engines compatible with the LLM model you plan to adopt, drawing on official documentation and community information. - Initial evaluation
Pick two or three strong candidates and run simple benchmark tests, evaluating inference speed, memory usage, ease of setup, and the like. - Detailed benchmarking
For candidates that performed well in the initial evaluation, run more detailed benchmarks. Testing under conditions close to your actual use case is essential. - Scalability testing
With future growth in mind, also test scale-out (distribution across multiple GPUs) and scale-up (support for larger models). - Overall assessment
Weigh performance, scalability, ease of use, cost, community support, and other factors comprehensively, then make the final decision.
The important thing is not to choose an inference engine simply because it is "the newest" or "the most popular." Selecting the engine best suited to your organization's specific needs and constraints is what leads to long-term success.
Engine Selection Guidelines by Use Case
Different use cases can call for different inference engines. Here are some typical cases and recommended choices
Case 1: An R&D environment for experimenting with a wide variety of models
- Recommended: QCT, TGI
- Why: Broad model compatibility and ease of setup take priority
Case 2: A high-traffic production service environment
- Recommended: TensorRT-LLM, vLLM, DeepSpeed
- Why: High inference speed and concurrent request capacity are required
Case 3: Operating with limited GPU resources
- Recommended: vLLM, DeepSpeed
- Why: Memory-efficiency features enable efficient operation even with limited resources
Case 4: A large-scale distributed environment for the enterprise
- Recommended: DeepSpeed, TensorRT-LLM
- Why: Although open source, both have strong vendor involvement, so vendor support can be expected; they also offer excellent capabilities for distributed inference across multiple GPUs and nodes
Summary and Next Steps
Selecting an inference engine is one of the key decisions in building an LLM service. Choosing the right engine lets you maximize performance, optimize costs, and lay a foundation that can accommodate future growth.
Each of the engines introduced here has its own strengths and weaknesses—there is no all-purpose engine that is optimal in every situation. We recommend selecting carefully based on your LLM model, hardware environment, and requirements, and running actual benchmark tests whenever possible.
In the next installment, Step 5, "Estimating the GPU Node Configuration", we will look in detail at how to design an actual GPU node configuration using the inference engine you have selected. We will learn how to estimate the number of GPUs based on the engine's characteristics, calculate memory requirements, and optimize costs.
See you next time!
Qualiteg: Your Partner for LLM/AI Security
At Qualiteg, our engineering team has hands-on experience designing LLM inference and serving infrastructure. We never treat an inference engine as a black box: our local-LLM consulting is grounded in deep knowledge of attention computation, KV cache behavior, and more. From core technology to AI market analysis—"Which configuration fits in VRAM?" "vLLM or Hugging Face—what criteria should drive the decision?" "What is the shortest path to adapting an existing model to our domain?" "When should we use open LLMs versus commercial LLMs?" "How do we build a secure local LLM setup?" "How do we choose local LLMs and GPUs?" "What about GPU data center demand and market forecasts?" "What about AI market forecasts?"—please feel free to reach out.
