Meta's next open-source language model, Llama 3, is scheduled for release in July and is intended to be on par with GPT-4.
According to The Information's sources, the new version of the model will be more responsive to the user, providing context for difficult topics rather than blocking tricky questions.
For words that can have a dangerous or harmless meaning depending on the context (e.g. "kill"), Llama 3 is said to be better at differentiating. With Llama 2, Meta took a more conservative approach to security, wanting to avoid a PR disaster at all costs.
Meta wants to put someone in charge of the model's tone and safety training to make the model's responses more nuanced. However, three key people in the field of AI safety have left Meta this month alone, The Information reports.
There are currently two camps when it comes to AI safety: One is the highly regulated approach, such as OpenAI's ChatGPT, GPT-4, and Google Advanced, where the AI model has clear guidelines on how to respond in a polite and politically correct manner.
Then there is the more unconstrained open-source movement and the contrasting program of X owner Elon Musk, who deliberately offers a chatbot that better represents conservative world views. Unfortunately for the target audience, Grok is still too "woke" and respects transsexuals, for example.
Llama 3 could be twice as big as Llama 2
Whether Llama 3 will again be a pure language model or multimodal, i.e., able to understand and generate images as well as text, has not yet been decided. With more than 140 billion parameters, the largest version of Llama 3 could be twice as large as Llama 2 with a maximum of 70 billion parameters.
However, this would still be only a fraction of the original GPT-4 Mixture-of-Experts model with 1.76 trillion parameters. It is not known how many parameters the GPT-4 models currently running ChatGPT have, but it is probably less. Parameters are no longer the only and most important measure of the expected output quality of AI models.
Fine-tuning of Llama 3 has not yet begun, according to The Information's sources. The final model is said to reach GPT-4 quality. Llama 2 was released in July 2023.
Overall, Meta still has big plans for generative AI and wants to stay true to its somewhat open-source strategy. Meta CEO Mark Zuckerberg is one of Nvidia's biggest customers and plans to have some 600,000 graphics cards in use for AI training by the end of the year, with the long-term goal of developing AGI, just like OpenAI. Meta is also developing its own AI chip called Artemis.
