OpenAI was working on a new AI model, codenamed Arrakis, that would make AI inference cheaper. However, the company stopped development due to quality issues.
Arrakis was supposed to be as capable as GPT-4, but cheaper to deploy because the model was partially designed using the so-called sparse principle. This means that only parts of the neural network are used to process input, while in traditional "dense models" the entire neural network is active. Google, for example, uses the sparse principle in its Pathways AI project.
Development of Arrakis started last fall, and training began in the spring. But OpenAI's development team quickly realized that performance was insufficient, reports The Information, citing two people familiar with the project.
For a month, the team worked on tweaks, then the leadership team killed the project, the sources said. It is unclear why the sparse principle worked well in early tests but failed for the larger model.
Shifting to "GPT-4 Turbo"
Despite the setback, OpenAI may incorporate its work on Arrakis into other models, such as the planned Gobi multimodal model. After the failed training attempt with Arrakis, OpenAI researchers have focused on developing a GPT-4 version that generates faster responses, according to The Information.
This could be the GPT-4 version "0613" that was introduced this summer, which is significantly faster than version "0314" that OpenAI started with in March. Some users have complained about a loss of quality in certain areas, which OpenAI refutes, but which has caused the company to make 0314 available via API for longer than originally planned.
OpenAI takes AI to the desert
Gobi, Sahara, and Arrakis: The new AI model prototypes are named after deserts, in a nod to the sparse principle and in the hope that the models will work more efficiently and save costs. OpenAI partner Microsoft in particular is said to have hoped that Arrakis would make the use of generative AI in its products less expensive.
Microsoft itself has since shifted resources to developing less expensive language models, as the high cost of generative AI threatens emerging business models, according to a report in the Wall Street Journal.
Previous leaks about Arrakis
OpenAI leaker FeltSteam had previously reported on Arrakis. The model was said to generate images and videos in addition to text, to have fewer hallucinations than GPT-4, and to be suitable as an autonomous agent. In addition, Arrakis would be partially trained with synthetic data. Release was planned for next year.
According to FeltSteam, Arrakis may be less a single model than a system for generating more efficient models. It may have helped in the development of GPT-3.5 Turbo.
In retrospect, the existence of multimodal Arrakis may explain a statement made by the CTO of Microsoft Germany, Andreas Braun. In early March 2023, one week before the official presentation of GPT-4, he spoke of "multimodal models" that could also generate video. At that time, OpenAI was probably still working on Arrakis.
