Content
summary Summary

According to a first rumor, Llama 3 will be able to compete with GPT-4, but will still be freely available under the Llama license.

Ad

This was overheard by OpenAI engineer Jason Wei, formerly of Google Brain, at a Generative AI Group social event organized by Meta. Wei says he picked up on a conversation that Meta now has enough computing power to train Llama 3 and 4. Llama 3 is planned to reach the performance level of GPT-4, but will remain freely available.

Image: Jason Wei via Twitter

Even if Wei himself is a credible source, the statements he heard could be wrong, or the plans could still change. There is no official statement on if or when Llama 3 will be released.

Meta took about five months between the release of Llama 1 in late February 2023 and the release of Llama 2 in late July 2023.

Ad
Ad

GPT-4 has a more sophisticated architecture than your standard Llama

GPT-4 likely achieves its high performance by using a more complex mixture-of-experts architecture with 16 expert networks, each with about 111 billion parameters.

Jumping from Llama 2 to Llama 3 may therefore be more challenging than simply scaling through more training, and may take longer than moving from Llama 1 to Llama 2.

Llama 2 reaches the level of GPT-3.5 in some applications and is also being optimized by the open-source community through fine-tuning and additional features.

For example, the recently released Code Llama, which is based on Llama 2, achieves GPT-3.5 and GPT-4 level results (depending on the type of measurement) in the HumanEval coding benchmark through fine-tuning.

However, in the paper on Llama 2, Meta itself notes that there is still a large performance gap with closed-source models such as GPT-4 and Google's PaLM-2.

Recommendation

The Financial Times reported in mid-July that the main goal of Meta's Llama models is to break OpenAI's dominance in the LLM market. Meta is likely trying to establish Llama models as an enabling technology in the LLM market, similar to what Google has done with Android in the mobile market, to launch additional offerings later. Meta also benefits from the rapid development of the models by the open-source community.

OpenAI chief Sam Altman said in early June 2023 that GPT-5 is still far from a training launch. Google plans to launch Gemini, the next generation of multimodal LLMs, late this year or early next year.

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • According to an early rumor, Meta is working on Llama 3, which is intended to compete with GPT-4, but will remain largely free under the Llama license.
  • OpenAI engineer Jason Wei reports that he has heard that Meta has enough computing power to train Llama 3 to the level of GPT-4. He also says that training Llama 4 is already possible.
  • While Wei himself is a credible source, his statements may be wrong, or the plans may change.
Sources
Online journalist Matthias is the co-founder and publisher of THE DECODER. He believes that artificial intelligence will fundamentally change the relationship between humans and computers.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.