Researchers present RoboGen, a generative robotic agent that automatically learns new skills in a generative simulation.
The work by researchers from CMU, Tsinghua IIIS, MIT CSAIL, UMass Amherst, and the MIT-IBM AI Lab aims to leverage recent advances in generative AI to generate infinite training data for automated robot learning.
According to the team, RoboGen is a generative robotic agent that learns various robotic tasks automatically and en masse through generative simulation. The team is using existing foundation models, such as OpenAI's GPT-4, to "automatically generate diversified tasks, scenes, and training supervisions, thereby scaling up robotic skill learning with minimal human supervision."
RoboGen uses GPT-4 for generative simulation
Specifically, RoboGen has an automated pipeline with four steps:
Task proposal: RoboGen selects a robot model and object and uses GPT-4 to generate tasks based on this information.
Scene generation: RoboGen creates simulation environments for the proposed tasks. This is done by selecting 3D objects from the Objaverse database and placing them in realistic arrangements using GPT-4.
Training supervision generation: RoboGen then uses GPT-4 to break down all the tasks into small steps. According to the team, this decomposition leads to less demanding tasks that can then be solved by algorithms such as reinforcement learning. After this step, GPT-4 selects an appropriate algorithm for each task.
Skill learning: Once the task, the scene in the simulation environment, the subtasks, and the algorithms have been selected, the scene and training in the simulation are started.
Video: Wang, Xian, Chen et al.
RoboGen to be developed further
In a first test, RoboGen generated training simulations for more than 100 different tasks. According to the team, the resulting dataset already outperforms human-generated datasets. However, the team did not compare their work with Google Deepmind's recently unveiled Open-X Embodiment dataset, which is also designed to support general robotics learning with different types of robots. Such a dataset could potentially be used in the future to improve RoboGens' capabilities.
Video: Wang, Xian, Chen et al.
The team cites the lack of verification of learned skills as a limitation of RoboGen, but automated processes will be integrated in the future. In addition, the reality gap remains a problem - but is narrowing as research progresses.
All information, further examples, and the code are available on GitHub.
