Understanding Mistral NeMo: Model Size, Context Window, and Tekken Tokenizer

Understanding Mistral NeMo: Model Size, Context Window, and Tekken Tokenizer

On July 18, 2024, Mistral AI announced the release of Mistral NeMo, a 12B parameter model developed in collaboration with NVIDIA. This release marked a significant step in bringing frontier AI capabilities to a broader audience, specifically targeting multilingual applications and enterprise adoption. The model is designed to be accessible, offering pre-trained base and instruction-tuned checkpoints under the Apache 2.0 license to facilitate use by researchers and enterprises alike.

Model Architecture and Deployment

Mistral NeMo is built on a standard architecture, which the publisher describes as making the model easy to use and a drop-in replacement for systems currently utilizing Mistral 7B. This architectural compatibility allows for straightforward integration into existing workflows without requiring extensive infrastructure changes.

The model weights are hosted on HuggingFace for both the base and instruction-tuned variants. For inference and adaptation, Mistral AI provides specific tools: users can try the model using mistral-inference and adapt it using mistral-finetune. Additionally, the model is exposed on la Plateforme under the name open-mistral-nemo-2407. For enterprise environments, the model is packaged as an NVIDIA NIM inference microservice and is available from ai.nvidia.com.

Context Window and Reasoning Capabilities

A defining feature of Mistral NeMo is its large context window, which extends up to 128k tokens. This capacity allows the model to process and retain information from extensive documents or long sequences of interaction.

In terms of performance, Mistral AI states that the model's reasoning, world knowledge, and coding accuracy are state-of-the-art within its size category. To validate these claims, the publisher provided a comparison of the Mistral NeMo base model against two other recent open-source pre-trained models: Gemma 2 9B and Llama 3 8B. The comparison, detailed in Table 1 of the announcement, highlights the model's competitive standing in accuracy metrics.

It is important to distinguish between the pretraining phase and the fine-tuning phase. The base model serves as the foundation, while the instruction-tuned version undergoes an advanced fine-tuning and alignment phase. This phase significantly enhances the model's ability to follow precise instructions, handle multi-turn conversations, and generate code compared to the earlier Mistral 7B model. Accuracy evaluations for the instruction-tuned model were conducted using GPT4o as a judge on official references, as noted in Table 2 of the source material.

Multilingual Proficiency

Mistral NeMo is explicitly designed for global, multilingual applications. The model is trained on function calling and demonstrates particular strength in a wide array of languages, including English, French, German, Spanish, Italian, Portuguese, Chinese, Japanese, Korean, Arabic, and Hindi.

The publisher notes that this development represents a new step toward bringing frontier AI models to everyone's hands in all languages that form human culture. Performance on multilingual benchmarks is illustrated in Figure 1 of the announcement, showcasing the model's capabilities across these diverse linguistic landscapes.

The Tekken Tokenizer

A critical component of Mistral NeMo's efficiency is the use of a new tokenizer named Tekken. This tokenizer is based on Tiktoken and was trained on over 100 languages. Its primary function is to compress natural language text and source code more efficiently than the SentencePiece tokenizer used in previous Mistral models.

The efficiency gains provided by Tekken vary by language type. Specifically, the tokenizer is approximately 30% more efficient at compressing source code, as well as Chinese, Italian, French, German, Spanish, and Russian. For certain languages, the compression efficiency is even higher: Tekken is 2x more efficient at compressing Korean and 3x more efficient at compressing Arabic compared to previous methods.

When compared to the Llama 3 tokenizer, Tekken proved to be more proficient in compressing text for approximately 85% of all languages. These compression rates are visualized in Figure 2 of the announcement. The use of a more efficient tokenizer contributes to the model's overall performance, particularly when combined with quantisation awareness.

Quantisation Awareness and Inference

Mistral NeMo was trained with quantisation awareness, a technique that enables FP8 inference without any performance loss. This feature is crucial for deploying large models in environments where computational resources are constrained, as it allows for faster inference speeds and reduced memory usage while maintaining the model's accuracy.

Performance Comparisons

To provide context on the model's capabilities, Mistral AI released a table comparing the accuracy of the Mistral NeMo base model with Gemma 2 9B and Llama 3 8B. This comparison serves to position Mistral NeMo within the current landscape of open-source models. Additionally, the instruction-tuned model's accuracy is benchmarked against Mistral 7B, demonstrating improvements in instruction following and code generation.

The publisher emphasizes that these performance claims are based on evaluations done with GPT4o as a judge on official references. It is noted that a model demonstration is not an independently verified measurement, and missing chart or table values do not definitively prove winners in all scenarios. However, the data provided by Mistral AI indicates a strong performance profile relative to models of similar size.

Conclusion

Mistral NeMo represents a significant advancement in open-source large language models, offering a 12B parameter architecture with a 128k token context window and robust multilingual support. Through the integration of the Tekken tokenizer and quantisation-aware training, the model achieves high efficiency and performance suitable for both research and enterprise applications. With its availability under the Apache 2.0 license and support for standard inference tools, Mistral NeMo provides a viable option for developers seeking to leverage state-of-the-art reasoning and coding capabilities in a scalable manner.

Source: Mistral NeMo | Mistral AI — https://mistral.ai/news/mistral-nemo/