- Grok-1 is now available as Open Source
Update from March 17, 2024:
As announced, Grok-1 is now fully Open Source with code and weights. xAI has released the architecture and weights of its 314 billion parameter Mixture-of-Experts model under the Apache 2.0 license. This is the largest Mixture-of-Experts model released as open source so far. Two out of eight expert networks (25%) are active for each input token. Mistral's successful Mixtral and GPT-4 are also based on this architecture.
Grok-1 is the unrefined base model from the pre-training phase completed in October 2023, which has not yet been optimized for specific applications. It is also likely to be less, or not at all, safety-oriented than comparable models because it has not been tuned with human feedback (RLHF). xAI does not comment on the training data, only mentions large amounts of text, and does not provide any ethical or safety guidelines.
xAI trained the model from scratch using its technology stack based on JAX and Rust. More information and an installation guide about Grok-1 is available on GitHub.
Original post from March 11, 2024:
Elon Musk plans to open source his AI chatbot, Grok, this week
Elon Musk's chatbot "Grok" is going open source this week, Musk writes on X, without specifying which components will be open source and to what extent. Grok is based on Grok-1, the first large language model from x.AI, Elon Musk's AI company. The model took about four months to develop, including two months of training. The training data is cut off in Q3 2023, the context length of the model is 8,192 tokens. In benchmarks, Grok-1 is on par with GPT-3.5, a level now reached by many open-source models. For the Grok chatbot implementation on X, Grok-1 is configured to provide answers to "spicy questions" that would be rejected by most other AI systems. Elon Musk's open-sourcing of Grok could be a reaction to his lawsuit against OpenAI, in which he criticizes OpenAI for its closed-source model policy.
