[ChatStream] Quickly Measuring Inference Speed Under Concurrent Requests

Hello, this is the Qualiteg Product Development Team.

This time, we ran a quick measurement of RakutenAI-7B-Chat's performance when handling five simultaneous requests.

Test Environment

• GPU: A5000
• LLM: RakutenAI-7B-Chat AWQ-8
• Inference environment: ChatStream v0.7.2
• Number of concurrent requests: 5

Since a video is probably easier to follow, we recorded the actual experiment. Please take a look.

As shown in the video, enabling the multitasking feature in the ChatStream UI lets you open multiple chat windows on the same screen.

The multitasking feature is designed for coordinating multiple LLMs to carry out work, but this time we took advantage of it to send simultaneous generation requests to the same model. We introduced a quick measurement using this feature previously as well, but this time we used theinput prompt synchronization feature, which made the process even easier.
(When you type an input prompt into one text box, the other text boxes are updated in sync.)

As for the results, as the video shows, we achieved an inference speed of roughly 70 tokens/sec with five simultaneous requests.

Given the design of the ChatStream UI, if you do not mind the windows getting narrower, you can increase the number of concurrent requests as much as you like, so it would be nice if load testing of inference speed were just as easy. In practice, however, browsers limit the number of sessions that can be open to a single origin at the same time, so 20 or 30 concurrent requests are simply not possible due to browser constraints.

For production-oriented load tests at 20, 30, or 100 concurrent requests, we recommend LLMLoad, the load testing tool for LLM services that we develop. We plan to cover it in a separate article.

[Reference] The single-request case

The experiment with only one request can be seen in this video.

With a single request, the inference speed was 75 tokens/s, slightly faster than with five simultaneous requests. Put another way, we found that throughput barely drops at around five concurrent requests.