Snowflake AI Research Introduces Arctic-SnowCoder-1.3B: A New 1.3B Model that is SOTA Among Small Language Models for Code

At our AI lab, we understand the importance of high-quality data in training code models. It's crucial to have clean, well-structured data to ensure that the models can accurately and efficiently handle real-world programming tasks. The challenge lies in acquiring abundant and high-quality data, as raw data often contains noise and irrelevant information. To address this, we have developed a refined pretraining approach that focuses on progressively refining data quality over three distinct phases. This approach has resulted in significant improvements in model performance. Our Arctic-SnowCoder-1.3B model, trained on 555 billion tokens, has outperformed larger models trained on over 1 trillion tokens. This highlights the importance of data quality over quantity in pretraining code models. In conclusion, our approach underscores the critical role of high-quality data in the pretraining of code models. We are committed to providing practical guidelines for future model development and invite you to connect with us for AI KPI management advice and continuous insights into leveraging AI. You can reach out to us via email at hello@itinai.com, or stay tuned for updates on our Telegram channel and Twitter.