In a letter to the company's board of directors, OpenAI researchers are said to have warned of an AI discovery that could pose a threat to humanity.
This was reported by Reuters, citing two sources familiar with the matter. The letter is also linked to Altman's firing, but is not the only reason, according to Reuters.
According to a source from The Verge, the board never received such a letter, which is why it played no role in Altman's firing. Reuters says it has not seen the letter. The Information reports not on the letter itself, but on the "Q*" breakthrough described in it.
Q-Star can solve simple math problems autonomously
The alleged letter is about the algorithm Q* (pronounced Q-Star), which is capable of solving grade school math problems autonomously. These math problems were not part of the training data set. OpenAI's chief scientist, Ilya Sutskever, is reportedly responsible for its development.
Some OpenAI researchers see Q-Star as a breakthrough on the road to artificial general intelligence (AGI), as its math skills could be a sign that the AI system has developed more general and human-like reasoning skills.
In their letter to the board, the researchers pointed out the opportunities and potential dangers of the systems, according to Reuters, for example, it could speed up scientific breakthroughs. They also raised concerns about whether OpenAI had taken the necessary safety precautions for such an AI system.
The researchers also referred to the work of an "AI Scientist Team", the existence of which has been confirmed by several sources, according to Reuters. The group was formed by merging the former "Code Gen" and "Math Gen" teams. It is investigating how existing AI models can be tweaked to improve their reasoning abilities and ultimately perform scientific work.
The two researchers behind Q-Star are said to be Jakub Pachocki and Szymon Sidor, building on the work of Ilya Sutskever. A demo of the system is said to have been circulating on OpenAI for several weeks.
Pachocki and Sidor left shortly after Altman was fired by Sutsekver, allegedly for safety reasons. But they returned with Altman. OpenAI co-founder Greg Brockman is said to be working on integrating Q-Star into OpenAI products.
"AGI has been achieved internally"
It is possible that OpenAI leaker "Jimmy Apples" was referring to Q-Star when he tweeted in mid-September that "AGI has been achieved internally". OpenAI CEO Sam Altman even repeated the statement on Reddit, but later edited his message, adding that an AGI "will not be announced with a Reddit comment" and that he had only picked up on the meme.
In early November, Altman said that current language models still need "another breakthrough" toward AGI, and that simply scaling systems isn't enough. It would require a system capable of discovering new physical phenomena.
Altman then made a cryptic statement at the APEC Summit 2023 on November 16, shortly before his dismissal. He had been in the room four times in the history of OpenAI when the team pushed "the veil of ignorance back and the frontier of discovery forward." The last time was a few weeks ago, Altman said. This may coincide with "Jimmy Apple's" AGI tweet.
The Information also reports that Sutskever launched a project called GPT-Zero in 2021. Here, a language model would be given more time and compute to generate an answer to create new academic discoveries. It is this codename "Zero" that OpenAI leaker "Jimmy Apples" now brings up in connection with the rumors surrounding the letter.
