Adobe Research and Northwestern University have developed Sketch2Sound, an AI system that turns vocal imitations and text descriptions into professional sound effects and atmospheres.
Adobe Research and Northwestern University have created an AI system that could transform how sound designers work. Called Sketch2Sound, the tool lets users create professional audio by humming, making sound effects with their voice, and describing what they want in plain text.
The system analyzes three key elements of vocal input: loudness, timbre (which determines how bright or dark a sound is), and pitch. It then combines these characteristics with text descriptions to generate the desired sounds.
Video: García et al, Adobe Research
What makes Sketch2Sound interesting is how it understands context. For example, if someone enters "forest atmosphere" and makes short vocal sounds, the system automatically recognizes these should become bird calls - without needing specific instructions.
The same intelligence applies to music. When creating drum patterns, users can input "bass drum, snare drum" and hum a rhythm using low and high notes. The system automatically places bass drums on the low notes and snare drums on the high ones.
Fine-tuned control for professionals
The research team built in special filtering technology that lets users adjust how precisely they want to control the generated sounds. Sound designers can choose between exact, detailed control or a more relaxed, approximate approach depending on their needs.
This flexibility could make Sketch2Sound particularly valuable for Foley artists - the professionals who create sound effects for films and TV shows. Instead of manipulating physical objects to make sounds, they could potentially create effects more quickly through voice and text descriptions.
The researchers note that spatial audio characteristics from the input recordings can sometimes affect the generated sound in unwanted ways, but it's working to solve this issue. Adobe hasn't announced when or if Sketch2Sound will become a commercial product.
