OpenAI's latest GPT4o model is probably a bit better

Interestingly, OpenAI says it would like to provide more details on how the model responses differ but cannot due to the lack of advanced research into methods for granularly evaluating and communicating improvements in model behavior.

When possible, OpenAI points out new capabilities and specific improvements and will continue to do so. Meanwhile, the team is constantly working to improve the model by adding good data, removing bad data, and testing new research methods based on user feedback and offline evaluations. This model update follows the same approach.

This situation illustrates the challenge OpenAI and other AI companies face in accurately quantifying and communicating model improvements, which are often subtle changes based on different data and experiments.

In addition, even news of small updates creates high expectations and speculation in the AI community, as everyone looks for evidence that the current GPT-4 level, which has been in place for about a year and a half, can be significantly surpassed.

OpenAI tries to manage these expectations by stressing that the latest model is not an entirely new frontier model, but an improvement of the existing GPT-4o. Rumors have been circulating for a few days about a possible imminent release of a much more powerful model based on "Project Strawberry."

In the chatbot arena, where people rate LLM chatbot performance without knowing which chatbot they are chatting with, GPT-4o has reclaimed the top spot, slightly ahead of Google Gemini 1.5. However, the validity of such tests is limited and can vary greatly depending on the task. Personal testing is the only way to determine the best option for individual needs.

For API users, the latest model snapshot via the API "gpt-4o-2024-08-06" supports structured outputs and has an extended maximum output of 16,384 tokens. The dynamic model "chatgpt-4o-latest" always refers to the current version of GPT-4o in ChatGPT, which is the latest model mentioned at the beginning. All models were trained with data up to October 2023.