Hello. In this post we introduce "Mistral NeMo 12B," the latest LLM from Mistral AI, released on July 19, 2024.

We will walk through the model's features and performance, and then try it out in an actual chat session using ChatStream.

What Is Mistral NeMo 12B?

Mistral NeMo 12B is the latest model developed by Mistral AI in collaboration with NVIDIA. It is released under the Apache 2 license, making it a highly permissive model that can be freely used, modified, and distributed.

Video Walkthrough

The content of this article is also covered in the following video, so please have a look as well.

Key Features

This model has three major strengths:

1. A large context size and strong inference performance
2. Multilingual capability
3. An efficient tokenizer

1. Large Context Size and Strong Inference Performance

Mistral NeMo 12B is a relatively small model at 12 billion parameters, yet it delivers strong performance even within its size category. Compared with Google's Gemma 2 9B and Meta's Llama 3 8B, its context window stands out in particular. Mistral NeMo 12B has a context size of 128,000 — sixteen times the size of the other two models.

2. Multilingual Capability

Mistral NeMo 12B excels at multilingual tasks. Beyond English, it performs well across a wide range of languages, including Japanese, French, German, Spanish, Italian, Portuguese, Chinese, Korean, Arabic, and Hindi.

• Score on the MMLU multitask language understanding benchmark: 68% (on par with GPT-3.5 Turbo's 69.8%)
• Score on JMMLU, which evaluates Japanese language understanding: 59%

3. Efficient Tokenizer

Thanks to the adoption of "Tekken," a new compression technique, the model processes natural language text and source code far more efficiently. For Japanese in particular, it achieves a 1.56x improvement in efficiency.

Hands-On Chat Experience

Let's try chatting with Mistral NeMo 12B using ChatStream.net.

You can try the chat yourself at the following URL.

https://chatstream.net/?model_id=mistral_nemo_instruct_2407&ws_name=chat_app

In the video above, we tried things like the following:

1. Asking about Mistral AI in Japanese
2. Having it answer the same question in English
3. Trying an answer in French
4. Asking questions about the movie "Titanic"
5. Asking about the films of director James Cameron
6. Asking about the characters in "Terminator 2"
7. Asking the same question in English and comparing the accuracy of the answers

As a result, we found that the answers in English were more accurate than those in Japanese.

Code Generation Capability

We also had it generate Python code for temperature conversion and list processing, and it produced accurate code along with careful explanations.

Model Architecture Details

Inference Environment

The inference environment we used this time is as follows.

• GPU: A5000
• OS: Ubuntu
• Inference engine: our original dynamic batching / continuous batching inference engine (known as the "classic engine")
• Inference server: ChatStream Server
• UI: ChatStream WebUI

Using the ChatStream SDK, we were able to publish a Mistral NeMo 12B chat environment to the internet in about 20 minutes.

Conclusion

Mistral NeMo 12B is a multilingual LLM that delivers strong performance despite its compact size. Its large context size and efficient tokenizer are especially notable, and it looks well suited to practical applications such as RAG.

For questions about generative AI, new businesses leveraging LLMs, or building LLM services, please feel free to contact us at Qualiteg.