Content
summary Summary

Generative AI can create photorealistic images, first models can generate videos. Bytedance and Alibaba are now showing models that generate photorealistic video clips of animated people.

Ad

Both companies use slightly different architectures, but both essentially use diffusion models that adapt images to a model that specifies the poses of, say, a dance sequence. MagicAnimate, from Bytedance and the Show Lab at the National University of Singapore, and Animate Anyone, from Alibaba and the Institute for Intelligent Computing, both generate short video clips of dancing people or cartoon characters from a reference image and a dance sequence.

Using methods such as ControlNet and temporal stability techniques, the videos achieve much higher consistency than other text-to-video or image-to-video models, beating the current best of the TikTok benchmark by almost 40 percent.

Both methods require only one image and one motion sequence to generate videos — this can be a real person, the Mona Lisa, or an AI-generated image. Bytedance MagicAnimate can even animate several people simultaneously.

Ad
Ad

Video: ByteDance

More opportunities for TikTok — and AI influencers

With these methods, AI influencers, which today mainly exist in the form of static images and AI-generated text, could soon also pick up on current TikTok trends or create other short video clips. In the future, Bytedance could also offer the models directly on TikTok for its customers.

The code for MagicAnimate is available on the project pages on Github. A demo for MagicAnimate is also available. The code for Animate Anyone should also be available on Github soon - the team wants to make some improvements before the release.

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.
Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • Bytedance and Alibaba are developing AI models that generate photo-realistic video clips of animated people.
  • The models developed, Bytedance's MagicAnimate and Alibaba's Animate Anyone, achieve a higher level of consistency than previous text-to-video or image-to-video models.
  • These AI models could expand the possibilities for AI influencers and could be offered directly on platforms such as TikTok in the future.
Max is managing editor at THE DECODER. As a trained philosopher, he deals with consciousness, AI, and the question of whether machines can really think or just pretend to.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.