According to a first rumor, Llama 3 will be able to compete with GPT-4, but will still be freely available under the Llama license.
This was overheard by OpenAI engineer Jason Wei, formerly of Google Brain, at a Generative AI Group social event organized by Meta. Wei says he picked up on a conversation that Meta now has enough computing power to train Llama 3 and 4. Llama 3 is planned to reach the performance level of GPT-4, but will remain freely available.
Even if Wei himself is a credible source, the statements he heard could be wrong, or the plans could still change. There is no official statement on if or when Llama 3 will be released.
Meta took about five months between the release of Llama 1 in late February 2023 and the release of Llama 2 in late July 2023.
GPT-4 has a more sophisticated architecture than your standard Llama
GPT-4 likely achieves its high performance by using a more complex mixture-of-experts architecture with 16 expert networks, each with about 111 billion parameters.
Jumping from Llama 2 to Llama 3 may therefore be more challenging than simply scaling through more training, and may take longer than moving from Llama 1 to Llama 2.
Llama 2 reaches the level of GPT-3.5 in some applications and is also being optimized by the open-source community through fine-tuning and additional features.
For example, the recently released Code Llama, which is based on Llama 2, achieves GPT-3.5 and GPT-4 level results (depending on the type of measurement) in the HumanEval coding benchmark through fine-tuning.
However, in the paper on Llama 2, Meta itself notes that there is still a large performance gap with closed-source models such as GPT-4 and Google's PaLM-2.
The Financial Times reported in mid-July that the main goal of Meta's Llama models is to break OpenAI's dominance in the LLM market. Meta is likely trying to establish Llama models as an enabling technology in the LLM market, similar to what Google has done with Android in the mobile market, to launch additional offerings later. Meta also benefits from the rapid development of the models by the open-source community.
OpenAI chief Sam Altman said in early June 2023 that GPT-5 is still far from a training launch. Google plans to launch Gemini, the next generation of multimodal LLMs, late this year or early next year.
