With Audiocraft, Meta releases three AI tools for music and audio generation for research purposes.
Audiocraft consists of Meta's MusicGen, an AI model introduced in June 2023 that can generate melodies and musical pieces from text and other music. Also part of Audiocraft is AudioGen, a Transformer-based generative AI model introduced in October 2022 that can generate sounds to match text input from scratch or extend existing audio files.
Meta's audio tokenizer EnCodec, which breaks audio files into smaller pieces for AI processing, is the third part of Audiocraft and is now available in an enhanced version that Meta says produces higher-quality music with fewer artifacts.
Model kit for AI audio experiments
According to Meta, the Audiocraft family of models can produce high-quality, consistent, and longer audio using only natural language interaction. The release provides full access to Meta's research in generative audio AI over the past several years, according to the company.
"There are nearly limitless possibilities once you give people access to the models to tune them to their needs," Meta writes.
With Audiocraft, musicians or sound designers, for example, would have professional tools for faster inspiration, brainstorming, or refining existing compositions.
MusicGen example: Earthy tones, environmentally conscious, ukulele-infused, harmonic, breezy, easygoing, organic instrumentation, gentle grooves
Audiogen example: Whistling with wind blowing
Generative audio to lower entry barriers to music and audio
The Meta research team continues to work on generative audio, specifically high-quality audio based on diffusion models, the same technique that has led to huge quality improvements in image generation.
The goal, for example, is to enable musicians to create new compositions without having to play a single note on an instrument or to help indie developers on a shoestring budget fill virtual worlds with believable and varied sound effects. For Instagram, generative audio AI could provide the right soundtrack for posts. However, Audiocraft does not yet allow commercial use, so this will not happen just yet.
Following the release, Meta once again stresses the importance of open-source models: "Responsible innovation can’t happen in isolation. Open sourcing our research and resulting models helps ensure that everyone has equal access."
Audiocraft's code is available here.
