Meta has developed a cutting-edge generative AI model ‘Voicebox’, designed to revolutionize the field of speech generation.
“We’ve developed Voicebox, the first model that can generalize to speech-generation tasks it was not specifically trained to accomplish with state-of-the-art performance,” Meta said in a blog post.
According to the company, Voicebox generates images and text in a variety of styles, and it can create outputs from scratch or modify samples provided to it.
However, instead of creating a picture or a passage of text, Voicebox produces high-quality audio clips.
The model supports speech synthesis across six languages, including English, French, German, Spanish, Polish, and Portuguese, as well as performs noise removal, content editing, style conversion, and diverse sample generation.
Moreover, Meta said that Voicebox uses a new approach to learn just from raw audio and an accompanying transcription.
Unlike autoregressive models for audio generation, Voicebox can modify any part of a given sample, not just the end of an audio clip it is given.
Further, the tech giant said that Voicebox is trained to predict a speech segment when given the surrounding speech and the transcript of the segment.
Once the model has learned to infill speech from context, it can be applied across a wide range of speech generation tasks, including generating portions of an audio recording without re-creating the entire recording.
This versatility enables Voicebox to perform well across a variety of tasks, including — in-context text-to-speech synthesis, cross-lingual style transfer, speech denoising and editing, and diverse speech sampling.
