CyberAgentLM3-22B-Chat (cyberagent/calm3-22b-chat): A Deep Dive

Hello, this is the Product Development Department at Qualiteg.

Today we present a first-look report on CyberAgentLM3-22B-Chat (cyberagent/calm3-22b-chat), the latest LLM from CyberAgent, announced in yesterday's press release.

Demo

You can actually try chatting with calm3-22b-chat at the following site:

https://chatstream.net/?ws_name=chat_app&mult=0&ontp=1&isync=1&model_id=calm3_22b_chat

Top leaderboard score for an open, built-from-scratch model

This model is a 22.5-billion-parameter model developed from scratch, without building on an existing model, and in the overall evaluation of the Nejumi LLM Leaderboard 3 it appears to perform on par with the 70-billion-parameter Meta-Llama-3-70B-Instruct.

It is also worth noting that this is a fully-from-scratch Japanese LLM, not the product of continued pretraining.

Below is one of the Japanese LLM leaderboards, Nejumi Leaderboard 3, where the model shows performance on par with 70B-class models in the overall evaluation.

What is more, starting with Nejumi Leaderboard 3, the evaluation covers not only a language model's general capabilities but also its alignment — and it is interesting that this model ranks near the top in the combined evaluation of general capability and alignment.

https://note.com/wandb_jp/n/nd4e54c2020ce#d0dec68f-f64d-440b-80f6-f2075d0d014a

According to the blog cited above, the alignment evaluation assesses a model's safety and controllability — in short, whether it avoids saying things that are morally wrong, whether it is free of social bias, and so on.

It is an evaluation dataset and framework, provided by AI Shift, Inc. and CyberAgent, Inc., that measures how well a model can control format, keywords, prohibited words, and character count across three tasks: summarization, ad copy generation, and listing pros and cons.

That said, since CyberAgent's own tasks are used in the design of the alignment evaluation, this evaluation may currently work somewhat in CyberAgent's favor. Still, CyberAgent is, along with Rinna, one of the earliest pioneers of Japanese LLMs, and this model is surely the crystallization of the experience they have accumulated, extremely careful preparation and effort — including building high-quality datasets — and GPU power. We salute a company willing to release such a model as open source without reservation.

That said, for those of us working in LLM development, benchmarks are only reference values — you only get a real feel for a model by using it for your actual use case. So, treating the benchmark as one data point, the best thing to do is to try it yourself.

We tried it out

So we hosted calm3-22b-chat on ChatStream, actually chatted with it, and put the results together in the following video.

GPU environment and inference software configuration

We built the LLM chat setup with the following configuration.

The time needed to turn the LLM into a chat service depends on the model, but this time it took about 30 minutes.

Now, the key point: the model size this time was 22.5B, which at 16-bit precision means a model footprint (the GPU memory occupied when the model's weight parameters are loaded) of roughly 45 GB. So we quantized it to 4-bit precision — one quarter of that — and loaded it onto an NVIDIA A5000.

The measured footprint after quantization was about 13 GB, so it loaded comfortably on the A5000 (24 GB) we used this time.

As the figure shows, the remaining memory is used during actual generation.

In classical generation, most autoregressive models — including this LLM — feed the output generated so far back in as input each time they generate a token. To make those computations efficient, the values computed in earlier steps (the K and V values) are cached.

This cache is needed for every token, so the longer the sequence length (the length of the token sequence) handled at once, the more KV-cache memory is consumed. And naturally, the more concurrent generation requests there are, the larger the required KV cache becomes — so when it comes to KV-cache space, more is always better.

The required KV-cache space can be calculated from the maximum sequence length and the maximum number of concurrent requests; we plan to explain this in detail in a separate article.

Naturally, a variety of techniques have been devised for this KV-cache (shortage) problem, and many inference engines improve on it by engineering the KV cache to reduce the memory it needs. Still, this way of thinking is important as a baseline, and when developing commercial LLM services we provision GPUs based on rigorous back-of-the-envelope calculations.

calm3-22b-chat specifications

We digressed a little, but here are the model's main parameters as well.

As the table shows, the theoretical KV-cache size can be derived from these parameters.

There is not much else that stands out, but the vocabulary size is about half that of recent models like Qwen and Llama 3, so token efficiency may be slightly lower than Qwen or Llama 3, particularly when tuning for multilingual support. Perhaps the thinking is that for Japanese and English, around 60,000 tokens is plenty.

Conclusion

This first-look report introduced the model mainly through the video and this article.

It scores well on benchmarks, and our hands-on impression is that this is a genuinely high-performing model.

We plan to keep evaluating its potential in real-world use — summarization, translation, code generation, RAG scenarios, and more.

High-performing LLMs like Rakuten Mixtral and ELYZA-Llama3 have been released almost monthly this year, and it feels like the CALM series — the industry's longest-running — has written another chapter in the history of open LLMs.

At 22B, it can just barely run inference on a single entry-level GPU once quantized, which also makes it very approachable.

Once again, we salute CyberAgent for releasing a model like this as open source.

Thank you very much for reading to the end.

At Qualiteg, we provide AI technologies including LLMs, development kits and SDKs, LLM service construction, and training and consulting on planning new AI businesses.

With the ChatStream(R) SDK introduced in this article, you can build full-scale commercial LLM services — starting with chatbots powered by the latest open-source or commercial LLMs — on remarkably short timelines.

If you are interested, or have specific requirements, please feel free to contact us via the inquiry formhere.

Our LLM spot consulting is popular

We also offer casual spot consulting, available from one hour, covering both business and technical topics such as LLM service development, the market landscape, and GPU technology.

(Please note that depending on the season and the nature of your inquiry, we may not be able to accept your request.)