Trying Out Codestral Mamba 7B, an Innovative Code-Generation LLM

Today we tried out "mistralai/mamba-codestral-7B-v0.1" (commonly known as Codestral Mamba 7B), a new code-generation LLM released on July 16, 2024.

This model adopts the new Mamba architecture and is published under the Apache 2 license.

Performance Approaching SOTA Code-Generation Models

Although Codestral 7B is built on the Mamba architecture, it achieves 75% on HumanEval, delivering performance on par with Transformer-based state-of-the-art code-generation models.

Moreover, since its processing does not degrade with sequence length, it is a very promising model and architecture.

Also Available as a Video

"mistralai/mamba-codestral-7B-v0.1" is also covered in this hands-on video report, so please have a look if you are interested.

Key Features of Codestral Mamba 7B

1. The ability to model sequences of unlimited length
2. Fast processing even on long sequences
3. Performance on par with the best Transformer-based models

What We Tested

1. Generating Python programs
   • Computing the sum from 1 to 1000
   • Printing the even numbers from 1 to 100
   • Generating the Fibonacci sequence
   • Converting Celsius to Fahrenheit
   • Generating random passwords
   • Extracting the second-largest number in a list
2. Code completion
   • A string reversal function
   • An average calculation function
   • A list flattening function
   • A BMI calculation function
3. Code generation from long-form instructions
   • Building an interactive chat using a Mistral LLM

Results

Codestral Mamba 7B produced satisfying output on every task. It generated accurate code when writing programs, and suggested appropriate implementations for code completion. It also handled long-form instructions flexibly, for example by generating code for an interactive chat that uses an API.

Conclusion

This post has been our first-look report on Codestral Mamba 7B.

Through generating a variety of Python programs and completing functions, we got a real sense of its strong performance and potential.

We have not yet put it through full-scale code generation, so next we plan to test it further from the standpoint of whether it can produce production-level code.