An experimental video shows what video games could feel like if characters were aware of and responsive to themselves and their surroundings.
YouTuber Foxmaster took on the classic game "Tomb Raider" in its original version. Using various AI tools for machine vision, localization, object recognition, animation, text, and speech, he breathed digital life into the game character, or more specifically, a Lara Croft bot that controls its own character.
First of all, it is not clear from the video to what extent the individual components of the project have been fully implemented. The description states that the video is "possibly inaccurate" and intended for entertainment purposes.
The gameplay passages where Lara Croft comments on the environment are definitely edited. According to Foxmaster, there is actually a longer pause between each of Lara's comments.
Still, as a thought experiment about the future of video games, the demo is worth a look. As Foxmaster describes it: "This is an example to make people realize the potential of these technologies combine."
ChatGPT as a personality engine
Lara Croft's personality is set by Foxmaster via ChatGPT using various attributes such as "brave", "friendly" or "smart". All objects that Lara perceives and comments on in the game pass through this personality filter and are reflected by Lara in the context of her given personality. For example, if she recognizes a sea lion, she will emphasize its swimming ability instead of making a comment about circus animals.
When it comes to object recognition, Foxmaster suggests a roundabout way, in part because the rudimentary computer graphics of the time required a lot of imagination to recognize a fern as a fern, for example.
However, a fern texture that is difficult to recognize still produces images of real ferns in Google image results, whose meta-information can then be fed into ChatGPT as input and produce a text that is suitable for Lara's voice synthesis to comment on.
The first 19 minutes of this video are dedicated to the technical theories behind the project, while the following video shows edited gameplay with an all-digital, self-aware Lara.
In particular, the use of ChatGPT with computer vision and object recognition is an interesting approach for future video games. Video game characters could be given a deeper and more flexible personality that is more responsive to their environment and what is happening in it. With today's video game characters, such interactions tend to follow predetermined rules and content.
A vivid example of this new flexibility is the use of speech synthesis models to interact with non-player characters that can respond to any kind of request from the player, rather than just predefined statements. This can improve immersion and even allow for new game mechanics.