Content
summary Summary

Update June 06, 2023:

Ad

Runway Gen-2 is now available in the browser and smartphone app for iOS. The trailer below shows some of the new features.

Original article dated March 20, 2023:

New York-based web video editing startup Runway introduces Gen-2, a new text-to-video model.

Ad
Ad

Runway first introduced its Gen-1 model in February, which can give existing videos a new look using only text prompts. For example, it transforms a realistically filmed scene into a cartoon world that retains the proportions and movements of the original scene.

This works for people and environments, and at a high level of abstraction: for example, Gen-1 can transform several notebooks standing upright next to each other into a skyline using only text prompts. All the Gen-1 features are included in Runway's new Gen-2 model.

Loosely assembled notebooks become a skyline. | Bild: Runway

Runway launches text-to-video model

Gen-2, however, takes it a step further by generating entirely new video scenes from a single text prompt. The following three-second video scene was generated by Runway with the prompt "Aerial drone footage of a mountain range". Audio is not yet included, but according to Runway, it is being researched.

Video: Runway

Prompt: "A close-up of an eye." | Video: Runway

Recommendation

In addition, Runway can generate short video sequences from an image or from the combination of an image and a text description. On the left in the screenshot you can see the input image. It's converted into a short video animation (on the right, enlarged in the screenshot). The background scene and camera angle change according to the text prompt.

Image: Runway

Bloomberg reports after a demonstration that the video generation happens "within minutes". But the resulting videos are only seconds long and a bit choppy. Motion sequences in particular are still a challenge for the model. Nevertheless, the generated scenes meet the content of the prompt.

Distribution via Discord waiting list

Runway is making Gen-2 available to select testers who sign up for a waiting list via Discord. The rollout is ongoing. Gen-1 currently has "thousands of users," according to Runway. The startup aims to prevent potential abuse of the video system, such as violent content, by combining AI mechanisms with the help of human moderators.

Gen-2-Trailer. | Video: Runway

Ad
Ad
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.

In addition to Runway, Google is working on an HD text-to-video AI system, and Meta demonstrated Make-a-Video, which also turns text into short videos. Google has another model in the works, Dreamix, specifically for text-to-video editing. If the rapid advances in AI image generation translate to video, stock video databases may soon have to rethink their business model.

Support our independent, free-access reporting. Any contribution helps and secures our future. Support now:
Bank transfer
Summary
  • New York startup Runway is developing a web video editor with AI capabilities.
  • Users have been able to edit videos with simple text prompts since the Gen-1 AI model.
  • The new Gen-2 goes one step further: It creates new videos from text alone. The model is now available to try for free.
Online journalist Matthias is the co-founder and publisher of THE DECODER. He believes that artificial intelligence will fundamentally change the relationship between humans and computers.
Join our community
Join the DECODER community on Discord, Reddit or Twitter - we can't wait to meet you.