Running Llama-3-Elyza-JP-8B with ChatStream🄬
Hello! Today we tried out Llama-3-Elyza-JP-8B.
Announced just yesterday, on June 26, 2024 (https://prtimes.jp/main/html/rd/p/000000046.000047565.html), Llama-3-Elyza-JP-8B is a Japanese LLM whose 70B version is reported to outperform GPT-4.
In this post, we share our experience loading Llama-3-Elyza-JP-8B into our own Playground environment and putting it through its paces.
Since the 70B (70-billion-parameter) version is said to surpass GPT-4, we decided to compare the 8B (8-billion-parameter) version against GPT-3.5.
(The performance comparison is described in detail in the article at https://note.com/elyza/n/n360b6084fdbd.)
Trying the AWQ-Quantized Version
This time, to build an inference server on an A4000—a relatively small GPU—we used the 4-bit AWQ-quantized model https://huggingface.co/elyza/Llama-3-ELYZA-JP-8B-AWQ.
Since the base model has roughly 8B (8 billion) parameters, 4-bit quantization brings the model size down to the equivalent of roughly 2B (2 billion) parameters.
The AWQ-quantized version is intended to run on vLLM as its inference engine, so we built our inference environment with vLLM 0.4.2 selected as the inference engine for ChatStream.
Deploying to ChatStream.net (Playground)
Using the ChatStream SDK, we created a single A4000 GPU server node for Llama-3-Elyza-JP-8B.
The setup took about 15 minutes.
All that remained was to register this server node with the front server of ChatStream.net, our Playground, and it was ready to go.
Including connectivity testing, the model was up and running in about 30 minutes total.
As you can see, deployment can be done remarkably quickly.
Measured with an LLM load-testing tool, it achieved stable throughput with almost no performance degradation up to 20 concurrent requests per second.
We expect it would hold up without issue to around 60 requests per second.
If you anticipate request volumes beyond that, the load can easily be distributed using ChatStream's model parallelism feature.
Try the Demo
"Llama-3-Elyza-JP-8B" vs. "GPT-3.5"
You can try Llama-3-Elyza-JP-8B yourself at the URL below.
Using ChatStream's multitask feature, GPT-3.5 is displayed alongside it for comparison.
(With the multitask feature and input sync, you can send the same question to multiple LLMs at once.)
Three-Way Comparison: "Llama-3-Elyza-JP-8B" vs. "RakutenAI-7B-chat" vs. "GPT-3.5"
To bring RakutenAI-7B-chat into a three-way comparison, add a model_id to the query as follows.
Architecture
The architecture we built for Llama-3-Elyza-JP-8B looks like this.
The server side of the ChatStream SDK is fully Dockerized, so
as long as you have a GPU server that can run the container, going from model preparation to public release takes about 30 minutes in total. Scaling out—with model parallelism, for example—takes anywhere from a few minutes to a few dozen minutes, so we can deliver the latest models to our customers right away.
Video
We put together a video covering everything described in this post.
Conclusion
Thank you for reading to the end. At Qualiteg, we provide AI technologies centered on LLMs, development kits and SDKs, LLM service construction, and training and consulting on how to plan new AI businesses.
With the ChatStream🄬 SDK introduced in this post, you can build full-scale commercial LLM services—starting with chatbots powered by the latest open-source or commercial LLMs—on remarkably short timelines.
If this has piqued your interest, or if you have specific requirements in mind, please feel free to reach out through the inquiry form here.
Our LLM Spot Consulting Service Is Popular
We also offer spot consulting, available casually from as little as one hour, covering both business and technical topics such as LLM service development, market conditions, and GPU technology.
(Please note that depending on the season and the nature of your request, we may not always be able to accommodate it.)
