Ad
Skip to content
Subscribe Now
AI in practice
Copy the url to clipboard Share this article Go to comment section

Snowflake's Arctic is the next open-source model focused on efficiency

Matthias Bastian
Matthias Bastian View the LinkedIn Profile of Matthias Bastian
Apr 29, 2024
Image description
Snowflake

Snowflake has developed its own large language model called Arctic and is now releasing it as open source. Arctic is designed to be highly efficient in both training and inference, especially for business-related tasks.

The company positions Arctic for enterprise applications, noting that the model excels at generating SQL code, general programming, and following complex instructions - capabilities that Snowflake groups under the self-defined metric of "enterprise intelligence".

According to Snowflake, Arctic required a training budget of less than $2 million, or about 3,000 GPU weeks. Despite this relatively low cost, the company claims that Arctic matches or exceeds the enterprise intelligence performance of larger models such as Meta's Llama 3 70B.

Image: Snowflake
Image: Snowflake

To achieve this training efficiency, Arctic uses a hybrid architecture that combines a Dense Transformer with a Mixture of Experts (MoE) residual layer. The base model is a Dense Transformer with 10 billion parameters, complemented by an MoE layer with a total of 480 billion parameters and 17 billion active parameters.

Snowflake hat die Metrik
Snowflake invented the "Business Intelligence" metric by combining some of the capabilities considered most important to businesses, such as SQL generation, and optimizing Arctic for these capabilities. On these metrics, it can match, if not outperform, models such as Meta's Llama 3 70B, which have been trained with a much higher budget. | Image: Snowflake

Snowflake has published a detailed "Cookbook" describing the model and its training process, and shares insights and best practices for training MoE models. The goal is to enable others to efficiently build large language models without extensive experimentation, the company says.

The model checkpoints for both the base and instructed versions of Arctic are now available for download on Hugging Face under the Apache 2.0 license. Instructions for inference and fine-tuning can be found on GitHub.

Snowflake also plans to work with Nvidia and the vLLM community to provide optimized implementations for fine-tuning and inference. The company is working on additional models in the Arctic series.

AI News Without the Hype – Curated by Humans

Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.

Read on for the full picture.
Subscribe for hype-free coverage.

  • Access to all THE DECODER articles.
  • Read without distractions – no Google ads.
  • Access to comments and community discussions.
  • Weekly AI newsletter.
  • 6 times a year: “AI Radar” – deep dives on key AI topics.
  • Up to 25 % off on KI Pro online events.
  • Access to our full ten-year archive.
  • Get the latest AI news from The Decoder.
Subscribe to The Decoder