Content
summary Summary

OpenAI was working on a new AI model, codenamed Arrakis, that would make AI inference cheaper. However, the company stopped development due to quality issues.

Ad

Arrakis was supposed to be as capable as GPT-4, but cheaper to deploy because the model was partially designed using the so-called sparse principle. This means that only parts of the neural network are used to process input, while in traditional "dense models" the entire neural network is active. Google, for example, uses the sparse principle in its Pathways AI project.

Development of Arrakis started last fall, and training began in the spring. But OpenAI's development team quickly realized that performance was insufficient, reports The Information, citing two people familiar with the project.

For a month, the team worked on tweaks, then the leadership team killed the project, the sources said. It is unclear why the sparse principle worked well in early tests but failed for the larger model.

Ad
Ad

Shifting to "GPT-4 Turbo"

Despite the setback, OpenAI may incorporate its work on Arrakis into other models, such as the planned Gobi multimodal model. After the failed training attempt with Arrakis, OpenAI researchers have focused on developing a GPT-4 version that generates faster responses, according to The Information.

This could be the GPT-4 version "0613" that was introduced this summer, which is significantly faster than version "0314" that OpenAI started with in March. Some users have complained about a loss of quality in certain areas, which OpenAI refutes, but which has caused the company to make 0314 available via API for longer than originally planned.

OpenAI takes AI to the desert

Gobi, Sahara, and Arrakis: The new AI model prototypes are named after deserts, in a nod to the sparse principle and in the hope that the models will work more efficiently and save costs. OpenAI partner Microsoft in particular is said to have hoped that Arrakis would make the use of generative AI in its products less expensive.

Microsoft itself has since shifted resources to developing less expensive language models, as the high cost of generative AI threatens emerging business models, according to a report in the Wall Street Journal.

Previous leaks about Arrakis

OpenAI leaker FeltSteam had previously reported on Arrakis. The model was said to generate images and videos in addition to text, to have fewer hallucinations than GPT-4, and to be suitable as an autonomous agent. In addition, Arrakis would be partially trained with synthetic data. Release was planned for next year.

Recommendation

According to FeltSteam, Arrakis may be less a single model than a system for generating more efficient models. It may have helped in the development of GPT-3.5 Turbo.

In retrospect, the existence of multimodal Arrakis may explain a statement made by the CTO of Microsoft Germany, Andreas Braun. In early March 2023, one week before the official presentation of GPT-4, he spoke of "multimodal models" that could also generate video. At that time, OpenAI was probably still working on Arrakis.

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • OpenAI stopped the development of Arrakis, an AI model that was supposed to be more efficient than GPT-4, because the quality was not satisfactory. Arrakis was based in part on the sparse principle, in which only parts of the neural network are activated, as opposed to the classic dense models.
  • Development started last fall and training began in the spring, but the team found that performance was insufficient. After a month of tweaking, the project was abandoned.
  • Despite this setback, OpenAI may incorporate the work on Arrakis into other models, such as the planned Gobi multimodal model. After Arrakis failed, the research team focused on a GPT-4 version that would generate faster responses.
Sources
Online journalist Matthias is the co-founder and publisher of THE DECODER. He believes that artificial intelligence will fundamentally change the relationship between humans and computers.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.