Improving Non-native Word-level Pronunciation Scoring with Phone-level
  Mixup Data Augmentation and Multi-source Information
  Deep learning-based pronunciation scoring models highly rely on the
availability of the annotated non-native data, which is costly and has
scalability issues. To deal with the data scarcity problem, data augmentation
is commonly used for model pretraining. In this paper, we propose a phone-level
mixup, a simple yet effective data augmentation method, to improve the
performance of word-level pronunciation scoring. Specifically, given a phoneme
sequence from lexicon, the artificial augmented word sample can be generated by
randomly sampling from the corresponding phone-level features in training data,
while the word score is the average of their GOP scores. Benefit from the
arbitrary phone-level combination, the mixup is able to generate any word with
various pronunciation scores. Moreover, we utilize multi-source information
(e.g., MFCC and deep features) to further improve the scoring system
performance. The experiments conducted on the Speechocean762 show that the
proposed system outperforms the baseline by adding the mixup data for
pretraining, with Pearson correlation coefficients (PCC) increasing from 0.567
to 0.61. The results also indicate that proposed method achieves similar
performance by using 1/10 unlabeled data of baseline. In addition, the
experimental results also demonstrate the efficiency of our proposed
multi-source approach.
