Meta has developed a new dataset that the company will use to improve the performance of automatic speech recognition (ASR) tools by clustering speech at the “utterance level”.
As part of Meta’s continued commitment to improving ASR performance, the company has taught ASRs to train without transcripts, recognize over 4,000 spoken languages, and even read lips more accurately than humans.
However, many of the datasets used to train ASR models are organized by demographic such as age group, gender, nationality, and English accent, which limits the variation of pronunciations that models are trained on, ultimately hampering their function in understanding a wide range of users.
In order to overcome this, Meta AI developed a dataset that relies instead on utterance clustering.
“Instead of dividing a dataset based on speakers’ demographic information — such as their age group or gender — our proposed algorithm clusters speech at the utterance level,” Meta said in a blog post on Thursday.
“A single cluster will contain similar utterances from a diverse group of speakers. We can then train our model using the various clusters and use fairness datasets to measure how the model impacts outcomes across different demographic groups,” it added.
The company’s resulting dataset includes about 27,055 utterances in a recorded speech by 595 people in the US who were paid to record and submit audio of themselves saying commands.
Their utterances are organized around seven main themes — music, capture, utilities, notification control, messaging, calling, and dictation, which other researchers can use to train their own models and digital assistants on.
The speakers were asked how they would voice search for a song, make plans with friends, and decide where to meet up.
To evaluate this new system, Meta trained their model on de-identified, publicly available Facebook videos in English which were evaluated on two datasets.
The first was a de-identified dataset collected from a data supplier for ASR that includes 48,000 utterances from 867 speakers, and the second dataset is Casual Conversations v1, a dataset of transcribed speech that Meta built and made publicly available in 2021.
“During testing, we observed that a model trained in this manner improved speech recognition accuracy for all measured demographic groups, and in particular for different accents, which are identified in sociolinguistics as a way of pronouncing a language that is distinctive to a country, area, social class, or individual,” Meta said.
“While our proposed algorithm was built using English-language data, we hope these approaches can be extended to work for other languages as well,” it added.
