An AI startup wants to use foundation models and computer vision to build bots that hunt bugs and gamers.
The Copenhagen-based startup is backed by several AI researchers, including Prof. Dr. Sebastian Risi from the IT University of Copenhagen, whose research includes lifelong learning for AI agents.
The Modl.ai team is using its expertise to develop intelligent bots that can test video games for bugs, performance, and other factors during development.
As video game worlds get bigger, like Bethesda's massive Starfield, and mechanics get more complex, traditional testing methods that rely solely on human labor are failing. Bots can help by moving around 3D worlds and collecting vital data.
The use of AI bots for testing and gaming has long been common. However, supervised-trained bots are currently the most widely used. The Modl.ai team instead relies on self-supervised trained bots. This is the learning method behind ChatGPT, for example, or today's best computer vision models.
Foundation models for testing and gameplay
The team places the intelligent bots in game environments without clearly defined data sets. The bot then learns from the available data to quickly evolve and act as an independent agent. In this way, the team aims to train so-called foundation models for AI agents.
These provide a foundation with many general capabilities on which developers can build for their specific applications, similar to how GPT-4 provides basic language processing capabilities that can be used for specific NLP purposes.
Similar to GPT-4, the team expects the Foundation models to provide rudimentary reasoning and planning skills. "The more complex the game, the more complex reasoning is required of bots to test it, and our team is tackling that issue head on," Risi said.
Foundation models could learn from gamers with computer vision
The team sees great potential in combining such foundation models with computer vision. "Recent advances in computer vision look at how a game is played by people and analyzes the common behaviors. Eventually, this research will enable a bot to look at a game it has never played before and know how to play it. Our research is driving this technology further and forward," says Georgios Yannakakis, one of the founders.
In addition to bots for testing, Modl.ai is also developing bots that can play with and against human gamers after release. Modl.ai comes with plugins for Unreal Engine and Unity, as well as an API. Should the team realize its vision, gamers can expect not only bug-free and better-optimized games, but also bots that are more than just cannon fodder.
Projects such as Voyager, MineCLIP, and VPT, which also use foundation models to power Minecraft bots, show that the approach has potential.
