Hello!

At Qualiteg, we operate AI-powered video conversion and video generation services. Services like these place heavy computational loads on GPUs during inference — that is, the processing that happens when a video is generated.

As a result, users have to wait for results while the GPU generates the video. Image and video generation in particular involves enormous amounts of computation, so wait times get long. The instant-response experience of conventional SaaS is hard to achieve, and waits can stretch from tens of seconds to several minutes, or even tens of minutes.

When designing such a server system, one question is always critical: "How many users can one server — or rather, one GPU — support?"

Accurate capacity calculation is especially essential for constrained systems, such as a conversion server that can process only one job at a time.

In this article, we explain how to calculate the optimal number of users for such a conversion server based on queuing theory. We also provide a practical calculator tool.

Problem Setup

Let's set up a simple problem

• There is a server that can process only one conversion at a time
• One conversion takes 5 minutes on average
• Users access the service at irregular intervals
• Access concentrates during peak hours

Under these conditions, how many users should a single server support in order to provide a comfortable service?

Fundamentals of Queuing Theory

This problem can be solved with queuing theory.

Queuing theory is a mathematical framework for analyzing how waiting lines behave when demand piles up against limited resources.

If we assume a single server, arrivals following a Poisson distribution, and service times following an exponential distribution, the system is what is known as an M/M/1 model.

Deriving the Calculation Model

First, let's define the basic parameters

• t_p: processing time (minutes/job)
  The average time the conversion server takes to complete one processing request. For a file conversion, for example, this covers the entire pipeline from reading the file through processing to output. The smaller this value, the higher the server's throughput. Example: 5 minutes/job
  
• μ (mu): service rate (jobs/hour) = 60 minutes ÷ t_p
  The maximum number of requests that can be processed per hour. It is obtained by dividing one hour (60 minutes) by the processing time per job (t_p). For example, if processing takes 5 minutes, μ = 60 minutes ÷ 5 minutes = 12 jobs/hour. This is the server's theoretical maximum throughput.
  
• ρ (rho): target system utilization
  The target value for server utilization, ranging from 0 to 1. Queuing theory recommends setting it around 0.7-0.8 for stable operation. As ρ approaches 1, the queue grows rapidly and the user experience deteriorates. Example: 0.8 (80% utilization)
  
• P_c: peak factor
  The ratio of access volume during the busiest period compared with the average. For example, P_c = 2.5 means peak-time access is 2.5 times the average. This value is derived from historical access logs or estimated from comparable systems.
  
• H: operating hours (hours/day)
  The total number of hours the system operates per day. For a 24/7 system this is 24 hours; for a system available only during business hours it might be, say, 12 hours from 8:00 to 20:00.
  
• P_a: access probability (probability of use per day)
  The probability that a registered user uses the system on a given day. For example, P_a = 0.3 means each user has a 30% chance of using the system on any given day. If every user uses the system every day, P_a = 1.
  
• F: number of conversions per user per day
  The average number of conversion jobs a user runs on a day when they use the system. For example, F = 2 means users run an average of two conversions on days they use the system. This value depends on usage patterns.
  

According to queuing theory, the utilization ρ must be less than 1 for the system to be stable. In practice, to keep wait times within a reasonable range, it is recommended to setρ = 0.7-0.8 or thereabouts.

Calculation Procedure

Total number of users one server can support

N = R ÷ r_u

Average number of requests per user per day

r_u = P_a × F

Total number of requests processable per day

R = λ_a × H

Average arrival rate that keeps operation stable even at peak times

λ_a = λ_s ÷ P_c

Average arrival rate under stable operation

λ_s = ρ × μ

Calculating the service rate

μ = 60 ÷ t_p

Putting these together, the final formula is as follows!

N = (ρ × μ × H) ÷ (P_a × F × P_c)

Based on this formula, we can calculate the optimal number of users a single server can support.

Since doing the math by hand is tedious, we have prepared a tool for you.

Optimal User Count Calculator

Use the tool below to easily calculate the optimal number of users for your server. As you adjust each parameter, the number of supported users updates in real time.

Practical Operational Considerations

The tool above uses a simplified setup and calculation model to aid understanding, but real-world system operation requires taking additional factors into account, such as the following

1. Peak-time analysis

Collecting and analyzing actual access patterns yields a more accurate peak factor. For example, you can analyze system logs to build an hourly access distribution and compute the peak factor as the ratio of the peak to the average.

2. Variability in processing time

Actual processing times are not constant — they fluctuate. If processing time varies widely depending on the conversion content, it is important to consider not only the average processing time but also the maximum and the variance.

3. Revisiting the access probability

User access frequency changes with the time of day, day of week, season, and how much buzz the service is generating. Collecting data over the long term to estimate access probability more accurately improves the precision of the calculation.

4. Accounting for user experience

When calculating server capacity, user experience matters as much as technical limits. For example, the design needs headroom so wait times do not grow too long — this is why the target system utilization is set at around 0.7.

Automatic Parameter Optimization: Applying Machine Learning

Building on the considerations above, the ideal is to keep optimizing continuously using data from actual service operation.

And when it comes to optimization, that is where machine learning — one of our specialties — comes in.

Data collection and monitoring

For optimal capacity calculation, continuously collect data such as the following

• Access logs: access patterns by hour, day of week, and season
• Processing time logs: the relationship between each conversion's execution time and its content
• Wait time data: time users spend waiting between entering the queue and the start of processing
• User behavior data: cancellation rate, retry rate, usage frequency, and so on

Parameter optimization with machine learning

Based on the collected data, the following parameters can be updated and optimized automatically with machine learning models

1. Dynamic prediction of the peak factor
   • Use time-series analysis (ARIMA, Prophet) to forecast future peak factors by day of week and season
     - ARIMA stands for Auto-Regressive Integrated Moving Average, a statistical model widely used for time-series forecasting.
     - Prophet is a time-series forecasting tool developed by Facebook that can produce forecasts efficiently even on large datasets.
     
   • Anomaly (spike) detection and automatic adjustment for special events, promotions, and unexpected viral surges
     
2. Processing time prediction models
   • Regression models that predict processing time from the characteristics of the conversion (video type, file size, and so on)
   • Learning processing-time trends along axes such as per-user and per-content
     
3. Personalizing the access probability
   • Fine-grained access probability models based on user segmentation
   • Building predictive models based on users' behavioral history
     
4. Automatic tuning of the optimal system utilization
   • Reinforcement learning to optimize the balance between user satisfaction and processing efficiency
   • Adjusting utilization using user feedback data

Implementation approach

From here it is the same as implementing any machine-learning-based optimization system: build a data pipeline and efficiently aggregate the various log data.
Feed the log data into weekly training runs to predict the variables above. Using the predicted variables, you can estimate with greater precision how many users a single GPU server can handle, and plan your server scale-out strategy — adding servers to keep pace with user growth.

These variables only become known once the service is actually in operation, so at the start — as in this article — they are set based on experience.

Conclusion

Capacity calculation for a conversion server is not simply a matter of "how many jobs per hour" — it requires a holistic view of user behavior patterns, peak-time concentration, and system stability.

The calculation model introduced in this article is one practical approach grounded in queuing theory, and we believe it can be applied to capacity planning for many GPU server systems with long-running jobs. That said, it remains a theoretical estimate, so when deciding actual system capacity, it is best to combine it with adjustments based on measured data and with performance testing.