Hello!

Today we begin a multi-part series on inference infrastructure provisioning — GPU provisioning in particular — for building your own LLM service in-house.

Introduction

As LLMs have evolved, the use of public LLMs such as ChatGPT and Claude has spread rapidly through the enterprise. Forward-looking companies, however, are moving beyond these general-purpose LLMs toward building dedicated, domain-specific models. Two important needs are driving this shift: stronger coverage of company-specific expertise, and information security.

More and more cases require handling company-specific expertise and confidential information that general-purpose public LLMs cannot accommodate, so the number of companies considering building their own LLMs and running them on their own servers is growing fast. In highly specialized fields such as finance, healthcare, manufacturing, and legal, industry-specific LLMs are increasingly seen as a source of competitive advantage.

Running an industry-specific LLM in-house, however, is no small feat. Once you decide to operate it yourself, the first thing to tackle is putting a proper inference environment in place. Whether you build on-premises or use a cloud service, you need to estimate in advance what hardware specifications are required — and in particular, what kind of GPUs you need and how many.

This is not simply a matter of buying GPUs; it requires careful calculation based on the service's expected load and the characteristics of the model. By properly estimating the type and quantity of GPUs required, you can accurately budget the procurement needed to secure the necessary performance while avoiding over-investment.

To build such a self-hosted LLM service effectively, proper "provisioning" of GPU resources and the inference environment is essential. Building an LLM environment that balances cost efficiency and performance without compromising the user experience is no easy task.

In this article, we explain the concept of "inference infrastructure provisioning" — the work required before an LLM service goes into production — and the concrete steps for carrying it out.

LLM Inference Infrastructure Provisioning Course: Series Index

• Part 1: Basic Concepts and Inference Speed
• Part 2: Estimating Request Volume for an LLM Service
• Part 3: Estimating Inference-Time Memory Consumption for Your Model
• Part 4: Selecting an Inference Engine
• Part 5: A Practical Process from GPU Node Configuration to Load Testing
• Bonus: KV Cache Offloading Strategies and GQA

What Is Inference Infrastructure Provisioning?

Inference Infrastructure Provisioning: The Key to Building an LLM Service

As more companies build and operate LLM services in earnest, securing the right hardware resources before moving to production has become a real challenge. Procuring high-performance GPUs and building an optimal inference environment are critical factors that determine both service quality and cost efficiency.

Preparing the GPUs and inference server resources (the inference environment) needed before an LLM service goes into production is what we call "inference infrastructure provisioning." This is a comprehensive process that covers not just securing GPUs but designing and preparing the entire inference infrastructure that supports the service.

An Overview of Inference Infrastructure Provisioning

Inference infrastructure provisioning means preparing the GPUs and inference server resources (the inference environment) required before an LLM service goes live in production.

The process consists of two main elements.

1. GPU Provisioning

In GPU provisioning, you determine the type and number of GPUs that satisfy the LLM service's requirements, based on the expected load of the service you plan to offer and the characteristics of the LLM model.

Concretely, this involves weighing factors such as:

• Projected number of concurrent users
• Response time requirements
• Number of tokens to be processed
• Hardware requirements of the selected LLM model
• Cost efficiency

Taking all of these into account, you derive the optimal GPU type (NVIDIA A100, H100, and so on) and the number of units required.

2. Inference Environment Provisioning

Inference environment provisioning covers choosing the servers, workstations, or cloud environment that will host the GPUs, deploying the software the service needs into the appropriate environment, and performing the necessary configuration.

The main tasks here are:

• Selecting inference server hardware or building a virtual environment
• Setting up a container environment (Docker, Kubernetes)
• Installing a model-serving framework (vLLM, TensorRT-LLM)
• Configuring load balancing
• Deploying a monitoring system
• Security measures

Qualiteg's 7-Step GPU Provisioning Process

With the importance of inference infrastructure provisioning established, let us introduce the concrete GPU provisioning process Qualiteg actually uses. At Qualiteg, we determine the type and volume of GPUs required for the target LLM service through the following seven steps.

Step 1: Define the Inference Speed

First of all, clearly define the inference speed you want to achieve. Inference speed is measured in "tokens generated per second" (tokens/sec). As an industry reference, OpenAI's ChatGPT runs at roughly 15-25 tokens/sec, and that level of speed is generally regarded as the minimum bar. Set a concrete target — for example, 25 tokens/sec.

Step 2: Estimate Request Volume for the LLM Service

Next, estimate the number of concurrent requests from the projected user request volume for the service. For services with pronounced peak and off-peak patterns, predicting the number of concurrent requests at peak time is especially important. For example, you might project 40 concurrent requests at peak.

Step 3: Estimate Inference-Time Memory Consumption for Your Model

Estimate the footprint of the model used by the LLM service (the amount of memory it consumes when loaded onto the GPU). You also need to account for the KV cache (the GPU memory consumed by text generation during inference). For example, a Llama3-8B model requires 16 GB plus the KV cache.

Step 4: Select an Inference Engine

Select an inference engine that supports your LLM model. An inference engine is software that performs the computations of inference efficiently. It is important to choose an engine optimized for your model; engines have GPU and platform dependencies, so avoid ones that do not support your setup. TensorRT is one representative example.

Step 5: Estimate the GPU Node Configuration

Calculate how many GPU nodes are needed to satisfy the LLM service's requirements. Building on the rough figure derived from the concurrent requests of Step 2 and the memory estimate of Step 3 via GPU request-time calculations, this step refines the estimate by factoring in GPU specifications and concurrency. It is common to draft multiple candidate configurations; since a cost cap is usually in place from the start, factor that in as well. One example would be a configuration such as A5000 × 4.

Step 6: Load-Test the Candidate GPU Configuration

Run actual load tests on the combination of the GPU configuration drafted in Step 5 and the selected inference engine. The memory consumption calculated in Step 3 is only a theoretical value, so it is important to apply real load and collect measured data. Tools such as Locust are often used for this load testing.

Step 7: Weigh the Trade-off Between Inference Experience and Cost

Finally, based on the data collected in Step 6, decide on the production GPU configuration and inference engine. Combinations with better numbers tend to require more expensive GPUs, so weigh the trade-off between the inference experience (chiefly inference speed) and cost. The outcome of this evaluation is ultimately compiled into an inference environment specification document.

Now, let's take a closer look at the key steps.

STEP 1: Defining Inference Speed — The Foundation of LLM Service Design

Synchronous and Asynchronous LLM Services

There are two patterns for delivering an LLM service: "synchronous" and "asynchronous."

What is a synchronous LLM service?

Services that demand real-time, immediate responses — chatbots, for example — fall into the synchronous category. When a user enters a question or instruction, an immediate response is expected. In this case, the faster the inference speed, the better the user experience of receiving the response — so when offering a synchronous LLM service, it is important to start by clearly defining the target inference speed.

A synchronous LLM service must anticipate a large volume of simultaneous requests, and handling them requires multiple GPUs and inference environments, so both the difficulty and the cost are higher than for the asynchronous type.

What is an asynchronous LLM service?

For use cases where immediacy is not required — for example:

• Summarizing large volumes of text
• Translating large volumes of text

— you can have the LLM run such jobs in batch and send a notification when the results are ready. This pattern is called an asynchronous service.

For an asynchronous LLM service where "finishing in a few hours or days is fine," a single GPU can sometimes handle the work efficiently, so the difficulty is lower and costs can be kept down.

Improving the Experience with Streaming Responses

The time from the user's input until the LLM's response appears on the user's screen is called "latency."

There are two types of response modes.

Single-batch response (all at once)

In this mode, text is generated from the input prompt and the response is sent only after all of the text has been generated. Nothing is returned to the user until the full result is ready, so the user is kept waiting. If the total latency is about 10 seconds, the entire result appears all at once.

Streaming response (incremental)

In this mode, text is generated from the input prompt and returned incrementally, one token (or a few tokens) at a time. The user can watch the text being generated, which makes for a better experience than an all-at-once reply.

With an initial-response latency of about 2 seconds, the first results start appearing; even if the complete-response latency is about 11 seconds, users can watch the output arrive bit by bit, so the perceived wait feels much shorter.

Inference Speed: Units and Target Setting

For a synchronous LLM service, the first priority is setting an inference speed target.

The unit of inference speed

Inference speed is expressed in "tokens per second" (written T/s or tokens/sec) — how many tokens can be generated in one second. It is the metric most commonly used to describe the performance of an LLM service.

The minimum inference speed to aim for

Being able to sustain 15-25 tokens/sec at peak request load is regarded as the minimum bar. In the subsequent steps, we will size the GPU configuration so that the inference speed set here can be achieved at peak time.

Comparing the Inference Speed of Major LLM Services

Comparing the inference speed of the major LLM chat services:

• ChatGPT (GPT-4): 15-25 tokens/sec
• ChatGPT (GPT-4o): 25-50 tokens/sec
• Claude 3 Haiku: 120 tokens/sec
• Groq (Llama 70B): 300 tokens/sec

A guideline for the inference speed to aim for

As a realistic target — taking ChatGPT's speed as the benchmark for end-user experience — something on par with GPT-4o, i.e. 40-70 tokens/sec, is a good guideline for the inference speed to aim for.

LLM inference speeds can be classified as follows.

• Slow (up to 30 tokens/sec): GPT-4 (15-25), DeepSeek-V2-Chat (25)
• Medium (30-100 tokens/sec): GPT-4o (40-70)
• Fast (100-300 tokens/sec): Claude 3 Haiku (120), Fireworks Llama3 70B (200)
• Ultra-fast (300+ tokens/sec): Groq Llama3 70B (302), Groq Llama3 8B (900)

As grading thresholds for your own LLM service, you might use 15 tokens/sec (grade C), 30 tokens/sec (grade B), and 70 tokens/sec (grade A).

These inference speed definitions form the foundation of the GPU provisioning process that follows, and directly affect the calculation of the GPU resources required.

Summary: The Road to Effective GPU Provisioning

In this installment, we covered the basic concepts of inference infrastructure provisioning for building an LLM service, along with a detailed look at STEP 1, "Defining Inference Speed" — a particularly important step in Qualiteg's 7-step GPU provisioning process.

Understanding the difference between synchronous and asynchronous LLM services and setting an appropriate inference speed target for your use case is the first step of provisioning. Using major LLM services such as ChatGPT and Claude 3 as reference points, we recommend setting a target in the range of 15-70 tokens/sec — a key metric that shapes the user experience.

Next time, we will cover STEP 2, "Estimating Request Volume for an LLM Service," and STEP 3, "Estimating Inference-Time Memory Consumption for Your Model," in detail: how to derive total GPU time and concurrent requests from DAU, how to calculate the KV cache, and how to run actual load tests and what to measure.

The GPU provisioning process is not just a technical calculation; it is a strategic approach to meeting business requirements and user experience goals. Securing the right GPU resources enables you to deliver a cost-efficient, stable LLM service.

See you next time!

Qualiteg: Your Partner for LLM/AI Security

At Qualiteg, our engineering team has hands-on experience designing LLM inference and serving infrastructure. We never treat an inference engine as a black box: our local-LLM consulting is grounded in deep knowledge of attention computation, KV cache behavior, and more. From core technology to AI market analysis—"Which configuration fits in VRAM?" "vLLM or Hugging Face—what criteria should drive the decision?" "What is the shortest path to adapting an existing model to our domain?" "When should we use open LLMs versus commercial LLMs?" "How do we build a secure local LLM setup?" "How do we choose local LLMs and GPUs?" "What about GPU data center demand and market forecasts?" "What about AI market forecasts?"—please feel free to reach out.

AI Technology Consulting | Qualiteg

Qualiteg Logo