Content
summary Summary

AI is supposed to make it possible for non-programmers to create complex programs like games. A group of researchers put this claim to the test by generating the game Flappy Bird 35 times using Python and ChatGPT.

Ad

The results of the DiverSE research group show that AI programming is not as easy as it is sometimes made out to be. But it has potential. The experiment doesn't provide a clear answer to whether non-programmers can suddenly just generate fully functional games with AI, but it does show that the process is neither simple nor solved.

Flappy Bird "Prompt Edition"

The study focused on developing the game Flappy Bird using Python as the programming language. The researchers explored several methods using ChatGPT 3.5 and 4, testing different prompts and strategies.

  • Writing a short prompt that simply describes the game,
  • Giving ChatGPT a more detailed list of features,
  • Providing a short description of the most important features,
  • Providing a complete code example and asking for a matching prompt, and
  • Using a series of prompts to generate code without reviewing the code in between.

However, the researchers did not find a magic prompt that always produced a playable game. Some trials produced playable games without additional technical intervention, but many sessions resulted in unusable games that required code fixes.

Ad
Ad

Interestingly, even with the same prompts, ChatGPT generated substantially different versions of code that produced completely different results. The team speaks of significant inconsistencies in the quality of the output, sometimes even leading to dead ends.

Amateur coding with ChatGPT? Yes, but…

The test shows that laymen with minimal technical knowledge can create games with ChatGPT, but that without programming knowledge this is hardly possible systematically. Often (but not always) direct intervention in the code was necessary to fix bugs.

If a game was in a bad state from the beginning, it was even harder to fix it without programming knowledge because of the lack of visual feedback. If you can't read code, you can only point out bugs that are visible in the game. But if the game does not work, this possibility disappears.

During the experiment, ChatGPT often split the problem and inserted placeholders into the code without updating the implementation. This is beneficial for the developers, but for the end user the game is incomplete and non-functional.

Overall, the team emphasizes the benefits of ChatGPT for programming, including inspiration for new variations of a game or unique features, the use of generated code as a starting point, and the basic ability for end users to create interesting games that sometimes work.

Recommendation

The team suggests further approaches, such as changing the programming language, finding better prompts or a language to better control ChatGPT, and improving the integration of ChatGPT output into the development environment to save time.

All 35 sessions with prompts, code, observations, and results using GPT-3.5 and GPT-4 via ChatGPT are available on Github. The video below shows some of the experiments and their results.

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • A team of researchers tested OpenAI's ChatGPT to see if non-programmers could use it to build complex game apps like Flappy Bird.
  • Despite mixed, sometimes impressive results, they could not find a magic prompt that generated a consistently working game. The team often had to dive right into the code to fix bugs.
  • Recommendations include testing more programming languages, finding better prompts with more control, and integrating ChatGPT more efficiently into the development environment.
Sources
Online journalist Matthias is the co-founder and publisher of THE DECODER. He believes that artificial intelligence will fundamentally change the relationship between humans and computers.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.