Content
summary Summary

Researchers present RoboGen, a generative robotic agent that automatically learns new skills in a generative simulation.

Ad

The work by researchers from CMU, Tsinghua IIIS, MIT CSAIL, UMass Amherst, and the MIT-IBM AI Lab aims to leverage recent advances in generative AI to generate infinite training data for automated robot learning.

According to the team, RoboGen is a generative robotic agent that learns various robotic tasks automatically and en masse through generative simulation. The team is using existing foundation models, such as OpenAI's GPT-4, to "automatically generate diversified tasks, scenes, and training supervisions, thereby scaling up robotic skill learning with minimal human supervision."

RoboGen uses GPT-4 for generative simulation

Specifically, RoboGen has an automated pipeline with four steps:

Ad
Ad

Task proposal: RoboGen selects a robot model and object and uses GPT-4 to generate tasks based on this information.

Scene generation: RoboGen creates simulation environments for the proposed tasks. This is done by selecting 3D objects from the Objaverse database and placing them in realistic arrangements using GPT-4.

Training supervision generation: RoboGen then uses GPT-4 to break down all the tasks into small steps. According to the team, this decomposition leads to less demanding tasks that can then be solved by algorithms such as reinforcement learning. After this step, GPT-4 selects an appropriate algorithm for each task.

Skill learning: Once the task, the scene in the simulation environment, the subtasks, and the algorithms have been selected, the scene and training in the simulation are started.

Video: Wang, Xian, Chen et al.

Recommendation

RoboGen to be developed further

In a first test, RoboGen generated training simulations for more than 100 different tasks. According to the team, the resulting dataset already outperforms human-generated datasets. However, the team did not compare their work with Google Deepmind's recently unveiled Open-X Embodiment dataset, which is also designed to support general robotics learning with different types of robots. Such a dataset could potentially be used in the future to improve RoboGens' capabilities.

Video: Wang, Xian, Chen et al.

The team cites the lack of verification of learned skills as a limitation of RoboGen, but automated processes will be integrated in the future. In addition, the reality gap remains a problem - but is narrowing as research progresses.

All information, further examples, and the code are available on GitHub.

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • Researchers present RoboGen, a generative robotic agent that automatically learns new skills in a generative simulation by generating infinite training data.
  • RoboGen uses foundation models such as OpenAI's GPT-4 in a four-stage pipeline that generates tasks, creates simulation environments, distributes training supervision, and finally learns robot skills.
  • In initial tests, RoboGen generated training simulations for over 100 different tasks and outperformed human-generated datasets. Future improvements will include verification of learned skills.
Sources
Max is managing editor at THE DECODER. As a trained philosopher, he deals with consciousness, AI, and the question of whether machines can really think or just pretend to.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.