Mistral Small v3 Explained: Now with Japanese Support
Hello!
On January 30, 2025, Mistral AI announced its new language model, Mistral Small v3. Despite a relatively small parameter count of 24B, the model achieves performance comparable to large-scale models of 70B or more. It also advertises Japanese-language support, making it a highly anticipated compact model.
https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501
Video
You can also watch our video walkthrough of this article here (in Japanese).
What Stands Out: Latency Optimization
The most striking feature of Mistral Small 3 is arguably its finely tuned balance between processing performance and latency.
According to the published performance evaluation chart below, it achieves an industry-leading latency of roughly 11 milliseconds per token. This is a clear advantage over Qwen-2.5 32B at about 15 milliseconds and Gemma-2 27B at about 14 milliseconds. Even more notably, compared with GPT-4o Mini, it delivers equal or better performance at lower latency.
(The measurements appear to have been taken with vLLM at batch size 16 on 4x H100.)
On the MMLU benchmark it achieves a high accuracy of 81% while delivering a throughput of 150 tokens per second. That amounts to more than a 3x speedup compared with Llama 3.3 70B Instruct, which offers comparable performance. Achieving both speed and accuracy is great news for building practical AI applications.
Comprehensive Evaluation and Demonstrated Advantages
In a rigorous third-party evaluation, comparative tests were conducted using more than 1,000 diverse prompts. Against Gemma-2-27B, Mistral Small 3 came out ahead in a full 73.2% of cases, broken down into 53.6% "clearly better" and 19.6% "slightly better."
Likewise, it outperformed Qwen-2.5-32B in 68% of cases, and even against the larger Llama-3.3-70B it was rated better in 35.6% of cases and equal in 23.6%, making it a model that shakes up the competition in the lightweight class.
On technical benchmarks, it achieves strong scores of 66.3% on MMLU Pro (5-shot CoT) and 45.3% on GPQA main. Particularly noteworthy are its 84.8% pass rate on HumanEval and 70.6% accuracy on math tasks. It also posts excellent results on instruction-following benchmarks—8.35/10 on MTBench and 87.3% on Arena Hard—demonstrating high reliability on practical tasks.
Now, from an everyday perspective, you might be wondering: how does it stack up against the much-talked-about DeepSeek R1?
To be fair, they are in different weight classes (DeepSeek R1 is a heavyweight, while Mistral Small is more of a lightweight), so a direct comparison isn't all that meaningful—but let's line them up anyway.
[Column] Comparison Table: The Trending DeepSeek R1 vs. Mistral Small 24B
While we're at it, let's compare it with the trending DeepSeek R1.
Below, we compare DeepSeek R1 (assuming the ~32B-scale version) and Mistral Small v3 24B across representative benchmarks and key specifications. Because the two models differ in design and scale, a strict comparison is not really appropriate. Please treat this only as a rough guide.
To keep the weight classes as close as possible, let's look at the DeepSeek R1 32B distilled version.
| Item | DeepSeek R1 (Distilled, ~30B version) | Mistral Small 24B (~24B) |
|---|---|---|
| Model size | ~30-32B (distilled version) * The original full model is MoE (several hundred B in total) | ~24B (dense Transformer) |
| License / openness | DeepSeek/MIT | Apache 2.0 (fully open) — anyone can download and use it commercially |
| MMLU (knowledge benchmark) | ~80% (estimated) * Estimate for the distilled version. The full R1 (MoE, hundreds of B) has been reported to exceed ~90% | Around 80-81%, comparable to 70B-class models |
| Code generation (HumanEval, etc.) | ~90% pass@1 (the distilled version also tends to be strong at code) * Needs verification on large-scale tests closer to real-world tasks | ~85% pass@1 — high marks on code benchmarks |
| Inference speed (reference) | - Around 30-40 tokens/s (A100 80GB, FP16) * Model size is somewhat larger. - The full version can be even slower | - Around 40-50 tokens/s (A100 80GB, FP16) - Focused on low latency through layer compression and speed-optimized implementation - Can be made even faster with 4-bit/8-bit quantization |
| Training data / methods | - Ultra-large-scale (~14.8T tokens) web/book/domain corpus - Uses RL (reinforcement learning) and MoE for higher accuracy (full version) | - Multilingual + code-centric - Instruction tuning only (no RLHF) - Strong character as a base model for general-purpose use |
Note: The figures, inference speeds, and benchmark scores above include estimates drawn from community and corporate reports. For official data, please consult the model providers' latest documentation.
Caveats
- The models belong to fundamentally different classes (in design philosophy and scale), so a precise comparison is difficult—but we are comparing them anyway, just for the fun of it. The full version of DeepSeek R1 falls into the ultra-large class at several hundred B parameters (MoE), whereas Mistral Small 24B is a mid-sized model aiming to be "lightweight and fast."
- Accordingly, the figures in the table above are merely a reference comparison against the DeepSeek R1 distilled version of a similar parameter scale. Ultimately, which model is the better fit depends on your deployment environment and task requirements.
Outstanding Multilingual Capabilities
Mistral Small 3's multilingual capabilities also deserve attention.
It shows excellent performance of about 73% in Japanese, and achieves high scores across European languages: about 78% in French, 77% in German, 78% in Spanish, and 75% in Russian. For Asian languages, it scores about 72% in Chinese and a somewhat lower 58% in Korean, but overall it demonstrates strong processing ability for non-English content.
This multilingual capability is underpinned by the adoption of the Tekken tokenizer, with its rich vocabulary of 131,000 tokens, and the implementation of a wide 32,000-token context window. These enable more natural and accurate language processing for complex context understanding and long-form text.
Implementation Architecture and Technical Innovation
Mistral Small 3's technical innovation is also evident in its architecture design. It achieves efficient processing in BF16 format and strengthens optimization for system prompts, enabling more natural dialogue and accurate responses. A standard deployment requires roughly 55GB of GPU RAM in BF16 or FP16 format, but quantization options make it possible to run in lighter-weight environments.
With quantization, it can even run on a single RTX 4090 graphics card or a MacBook with 32GB of RAM. These practical requirements add yet another option for small and mid-sized organizations and individual developers to leverage a high-performance language model.
Platform Availability and a Rich Development Ecosystem
The model is already available on our own ChatStream as well as major platforms such as Hugging Face, Together AI, and Fireworks AI, so developers can easily incorporate it into their projects. Availability on major cloud platforms including NVIDIA NIM, Amazon SageMaker, Groq, Databricks, and Snowflake is expected to follow shortly.
Locally as well, it is compatible with easy-to-use engines such as vLLM—an important feature that makes integration into existing AI projects straightforward.
Conclusion
While the LLM community is abuzz with DeepSeek R1, Mistral Small 3 has been steadily improving as well, and it may prove to be an important milestone in bringing lightweight, high-performance AI models into practical use. Its native support for Japanese is an especially big plus for those of us in Japan!
With our ChatStream platform, you can deploy Mistral Small 3 with no code and easily build a full-fledged commercial LLM service. Please feel free to contact us anytime about commercial development using Mistral Small 3!
