[Explainer] What Is the Tekken Tokenizer? Inside the New-Generation Tokenizer Adopted by Mistral

Hello!

Today we are going to talk about Tekken!

When you hear "Tekken," what comes to mind?

The fighting game Tekken (鉄拳, "iron fist"), perhaps?

Personally, I thought of the Long Swordsman (鉄剣戦士, "iron-sword warrior" in the Japanese edition) from Age of Empires, a game I played long ago 🤗
A bit dated, perhaps, but a true classic!

Now, enough with the icebreaker...
As you know, LLMs are evolving at an astonishing pace. Amid all this, one area quietly drawing attention is improvements to the tokenizer.

For example, the tokenizer in Meta's Llama models has evolved from SentencePiece to a BPE-based approach; across the LLM industry, more efficient tokenization (text segmentation) methods are being adopted.

Mistral AI, too, has adopted a new mechanism called the "Tekken tokenizer" to raise the performance of its large language models.

In this article, we will walk through the background behind the Tekken tokenizer, its technical characteristics, how it differs from other tokenizers, and its relationship with Mistral in an accessible way.

1. Background: Why the Tekken Tokenizer Emerged

1-1. Mistral AI and the challenge of very long contexts

Mistral AI (hereafter Mistral) is one of the most closely watched startups in the LLM industry. Following its earlier models (e.g., Mistral 7B), it has been releasing series that support large context lengths, such asMistral NeMo. Mistral NeMo in particular is notable for its enormous context length of128k.

Mistral Small 3, announced recently (January 30, 2025), also has a 32K context.

When working with contexts this large, what becomes critically important is increasing the amount of information carried per token. If the tokenizer is inefficient, the actual input text becomes "bulky," and even a 128k-token context cannot be used to its full potential.

So, in place of conventional SentencePiece or BPE (Byte-Pair Encoding), Mistral developed and adopted the Tekken tokenizer, achieving more efficient tokenization.

1-2. Release timing

The Tekken tokenizer was first introduced to the public in July 2024, announced together with the release of the models co-developed by Mistral AI and NVIDIA (commonly known as the Mistral NeMo series).

2. Technical Characteristics of the Tekken Tokenizer

2-1. BPE-based + multilingual

The Tekken tokenizer is a form of what is known as subword segmentation. It is built on OpenAI's tiktoken and adopts Byte-Pair Encoding (BPE), a scheme that segments strings across many natural and programming languages with high efficiency.

• A large-scale multilingual corpus was used for training, with more than 100 languages supported
• Source code and a wide variety of text containing special characters are also handled

It is designed to be especially strong in languages other than English: Japanese, Korean, Chinese, Arabic, and other language families can be represented with fewer tokens than with conventional tokenizers.

2-2. Large vocabulary and high compression

Conventional LLM tokenizers (for example, those of SentencePiece-based LLaMA models) often have vocabulary sizes of around 30,000 to 60,000.
The Tekken tokenizer, by contrast, has a very large vocabulary of roughly 130,000 words, and by also including more than 1,000 control tokens, it can handle over 130,000 tokens in total.

The advantage of a larger vocabulary is improved compression (how many tokens a single word is split into). Rare words, long proper nouns, programming-language keywords, and the like can be treated as single tokens, so the tokenized sequence becomes shorter. The result: those 128k tokens can hold much more actual text.

2-3. Special tokens (control tokens)

A notable aspect of the Tekken tokenizer is that it reserves roughly the first 10 to 14 tokens as control tokens.

• <unk>(unknown), <s>(beginning of sequence), </s>(end of sequence), and other standard tokens
• "[INST]", "[TOOL_RESULTS]", "[/INST]", and other special tags that Mistral uses inside prompts

By explicitly inserting these control tokens at the prompt-design stage, you can interact with the model while preserving the structure of the prompt. The design also includes mechanisms that help defend against prompt injection and manage prompts for tool execution, giving it a more advanced role than an ordinary tokenizer.

3. How It Differs from Other Tokenizers

The Tekken tokenizer draws attention because of its outstanding token efficiency and its multilingual versatility.
SentencePiece, long a common choice in earlier models, and the BPE used by GPT-family models are both perfectly capable, but Tekken is said to hold the following advantages.

1. High compression
   • For example, reported results show tokenization roughly 1.5 to 2 times more efficient for Japanese and 2 to 3 times more efficient for Arabic.
2. Large vocabulary
   • Covering roughly 130,000 words, it can process technical terms and mixed-language text without splitting them too finely.
3. Control tokens built in as standard
   • They help manage prompt structure and clarify conversational context, underpinning the implementation of safe dialogue flows rather than mere word segmentation.

Of course, the larger a tokenizer's vocabulary, the higher the training cost may become, so bigger is not automatically better. For models that handle ultra-long contexts like Mistral's, however, the total number of tokens consumed and the density of information carried per token both improve, so the benefits are considerable.

4. So Is Tekken the Best Tokenizer? → The Answer Is No

After reading this far, you might conclude that Tekken is simply the best—superior to SentencePiece—but that is not the case.

SentencePiece is not the only engine out there; the real question is how efficiently you segment the language you are working with. If you are building a Japanese-specialized LLM, for example, it is entirely possible for a SentencePiece vocabulary of around 32,000 tokens to be more efficient than Tekken with its 130,000 multilingual tokens.

Measuring Actual Token Counts

While we're at it, let's actually tokenize some text.

# Install the required libraries
# !pip install transformers sentencepiece  # Uncomment if you want to use Colab

from transformers import AutoTokenizer
from huggingface_hub import login

# Log in to Hugging Face
login(token="hf_xxxxxxxxxxxxxxxxxxx")

# Mistral Nemo (Tekken) tokenizer
mistral_tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-Small-24B-Instruct-2501")

# Llama 3 tokenizer
llama_tokenizer = AutoTokenizer.from_pretrained("tokyotech-llm/Llama-3-Swallow-70B-Instruct-v0.1")

# rinna tokenizer (SentencePiece-based)
rinna_tokenizer = AutoTokenizer.from_pretrained("rinna/japanese-gpt-neox-3.6b")

# Test text (Japanese)

text = """
人工知能は急速に進化しており、自然言語処理や機械学習の分野で革新的な成果を上げています。
特に大規模言語モデルの発展により、人間のような自然な対話や文章生成が可能になってきました。
"""
print("=== Processing with Tekken (Mistral Nemo) ===")
# Tokenize and get the ID sequence
tokens_tekken = mistral_tokenizer.encode(text)
print("Token IDs:", tokens_tekken)
print("Token count:", len(tokens_tekken))  # Show the token count

# Converting the ID sequence straight back to token strings exposes "partial bytes", which looks like garbled text
decoded_tekken = mistral_tokenizer.decode(tokens_tekken, skip_special_tokens=True)
print("String after decode():", decoded_tekken)

print("\n=== Processing with rinna (SentencePiece) ===")
tokens_rinna = rinna_tokenizer.encode(text)
print("Token IDs:", tokens_rinna)
print("Token count:", len(tokens_rinna))  # Show the token count
# With SentencePiece, the token strings are relatively readable as-is
tokens_rinna_decoded = rinna_tokenizer.convert_ids_to_tokens(tokens_rinna)
print("convert_ids_to_tokens:", tokens_rinna_decoded)
decoded_rinna = rinna_tokenizer.decode(tokens_rinna)
print("String after decode():", decoded_rinna)

Execution results

As you can see, in the case of the Japanese-specialized rinna model (rinna (e.g., japanese-gpt-neox-3.6b) is SentencePiece-based), token efficiency is simply better than Tekken's.

=== Processing with Tekken (Mistral Nemo) ===
Token IDs: [1, 1010, 3405, 26247, 7422, 15928, 2312, 36783, 42135, 2650, 38500, 23403, 5013, 48267, 1749, 43090, 7565, 15199, 5747, 1166, 12894, 5115, 23496, 28883, 1176, 12104, 12904, 1146, 2439, 5020, 23585, 2701, 38980, 11795, 2713, 2768, 5862, 9890, 4187, 66142, 2973, 29062, 1844, 50775, 5368, 47960, 86061, 7565, 15199, 24222, 69030, 2439, 9045, 60288, 74112, 1749, 113283, 2439, 98915, 43090, 2768, 22949, 8888, 5115, 117160, 7360, 5862, 3322, 31470, 52139, 6409, 8294, 25004, 1844]
Token count: 74
String after decode(): 
人工知能は急速に進化しており、自然言語処理や機械学習の分野で革新的な成果を上げています。
特に大規模言語モデルの発展により、人間のような自然な対話や文章生成が可能になってきました。


=== Processing with rinna (SentencePiece) ===
Token IDs: [263, 30008, 271, 16351, 8152, 1041, 264, 1770, 1920, 3001, 296, 2483, 3744, 16174, 13952, 618, 9655, 15104, 18732, 265, 263, 1085, 9273, 1920, 1120, 8824, 364, 264, 1609, 1976, 1770, 334, 17585, 296, 9572, 5195, 5778, 3642, 454, 5736, 265, 3]
Token count: 42
convert_ids_to_tokens: ['▁', '人工知能', 'は', '急速に', '進化', 'しており', '、', '自然', '言語', '処理', 'や', '機械', '学習', 'の分野で', '革新', '的な', '成果', 'を上げ', 'ています', '。', '▁', '特に', '大規模', '言語', 'モデル', 'の発展', 'により', '、', '人間', 'のような', '自然', 'な', '対話', 'や', '文章', '生成', 'が可能', 'になって', 'き', 'ました', '。', '</s>']
String after decode(): 人工知能は急速に進化しており、自然言語処理や機械学習の分野で革新的な成果を上げています。 特に大規模言語モデルの発展により、人間のような自然な対話や文章生成が可能になってきました。</s>

Llama 3 (e.g., Swallow 70B), for its part, uses not SentencePiece but the GPT-lineage Byte-Pair Encoding (BPE→tiktoken-based). Here too, the tokenization efficiency a tokenizer delivers varies with the situation, so there is little point in a side-by-side debate over which of Tekken, Llama 3, or SentencePiece is the superior tokenizer. (Debating tokenizer efficiency in isolation is of limited value in the first place.)

5. Why Mistral Adopted It, and How It Is Used

5-1. Better multilingual model performance

Mistral invests heavily in multilingual support, aiming for strong performance not only in English but also in Japanese, Chinese, Korean, and other languages. For that, appropriate tokenization for each language is essential. Tekken's high compression and versatility truly shine in multilingual models.

5-2. Making the most of very long contexts

As noted above, Tekken reduces token counts, so the same context length can hold far more actual text. Mistral NeMo supports 128k tokens, which is a major advantage in use cases that process much longer texts, documents, or source code in a single pass (e.g., document analytics and code review).

5-3. Safe, flexible prompt structure

The Tekken tokenizer has control tokens for structuring prompts built in. This matters especially for what Mistral plans to pursue going forward: leveraging agent capabilities and tool calling.

When an agent calls an external service and receives the result, the benefit is that text delimited by special tokens such as"[TOOL_RESULTS]" can be handled safely and reliably by the model. This strengthens resistance to prompt injection and helps keep user input and tool output from being conflated.

6. Conclusion: The Evolution the Tekken Tokenizer Brings

The Tekken tokenizer brings together:

• Multilingual and code support
• High compression and a large vocabulary
• Safe, flexible prompt structure via control tokens

It is a new-generation tokenizer combining all of these characteristics. It fulfills Mistral's goal of packing as much information as possible into each token when exploiting large contexts, and it builds in mechanisms that make prompt structure easy to leverage—for a future in which conversational AI plays a central role.

Going forward, we may well see models and frameworks beyond Mistral adopt Tekken too.

Tekken aside, more efficient tokenization is one part of improving model performance, so it will remain a hot topic in the large language model industry!

For LLM/AI security, talk to Qualiteg

At Qualiteg, our engineering team has hands-on experience designing LLM inference and serving infrastructure. Rather than treating the inference engine as a black box, we support local LLM adoption with deep expertise in areas such as attention computation and KV cache behavior. From core technology to AI market analysis—"Which configuration fits in VRAM?", "vLLM or Hugging Face: what are the decision criteria?", "What is the shortest path to adapting an existing model to our domain?", "When should we use open versus commercial LLMs?", "How do we build a secure local LLM configuration?", "How do we choose local LLMs and GPUs?", "GPU data center demand and market forecasts", "AI market forecasts"—please feel free to consult us.

AI Technology Consulting | Qualiteg

Qualiteg Logo