The AI model NeuralReshaper makes heavy people light, tall ones small or petite ones wide. It shows how quickly AI is making its way into the graphics field - and raises questions about how this affects our everyday use of visual media.
The work, "Single-image Human-body Reshaping with Deep Neural Networks," comes from researchers at Zhejiang University in Hangzhou and the School of Creative Media at the City University of Hong Kong.
In their paper, they describe the "NeuralReshaper," an AI model trained with Deep Learning that can transform the proportions of people in images in a fully automated way, regardless of the background. Among other things, the researchers use the GAN technology known from Deepfakes.
NeuralReshaper: AI tool reshapes human bodies
First, a skinned multi-person linear model (SMPL) generates a 3D estimate of a person's entire body in a photo based on a 2D photo. The SMPL model is trained with thousands of 3D scans of people. The Max Planck Institute for Intelligent Systems developed it in 2015 together with special effects company Industrial Light and Magic primarily for more efficient movie effects.
In the next step, the SMPL model modifies the 3D model generated from the 2D photo in the three attributes of height, weight and body proportions and performs the corresponding visual transformations. A person can control this reshaping process using simple sliders.
In the second step, "neural reshaping" takes effect. Here, foreground and background are separated and, among several other steps, a GA network generates a new 2D image of the person based on a template created in step 1 and inserts it into the background without distortions or overlaps.
The method works for both fully clothed and less clothed people. Even visually complex scenes, such as a person on a bicycle, are said to be easily modified by neural reshaping. In their paper, the researchers show many example images, proving that their method works robustly with many complex motifs - in other words, it is suitable for everyday use.
One of the researchers' innovations is the training method for generative image AI. Ideally, it would be trained with before-and-after image pairs of the same person.
Since such data hardly exist, if at all, the researchers developed a supervised training strategy in which a deformed version of the original image serves as input: during AI training, the neural network must try to reconstruct the original image from the deformed input image. In this way, it learns the visual difference between, for example, tall and short people regarding a single image of a person.
AI graphics processing is moving fast
"NeuralReshaper is the first deep-learning based method for realistic human reshaping in single images. Our tool is easy-to-use, fully automatic, and works well for images taken in unconstrained environments. Our method enables users to reshape human images by moving several sliders with instant feedback," the researchers write.
For now, the NeuralReshaper is just another tool for AI graphics that shows how profoundly artificial intelligence will change the graphics industry in particular. Digital reshaping of bodies has long been common practice, for example in the fashion industry. But with tools like NeuralReshaper, the process becomes so simple and fast that anyone can do it.
This accessibility brings opportunities, for example in therapeutic approaches, and risks when such sliders appear on Instagram or TikTok. Or, much more likely, the images modified with such sliders will find their way onto corresponding platforms - and in large numbers, rather than only on selected accounts supported by Photoshop professionals.
The increasing distortion of reality possible from such a development is likely to further intensify the debate about the effects of social media on the psyche, especially of young people.
