{'fraction_non_alphanumeric': 0.029411764705882353, 'fraction_numerical': 0.0, 'mean_word_length': 5.653631284916201, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Semi-supervised training strategies for deep neural networks

Use of both manually and automatically labelled data for model training is referred to as semi-supervised training. While semi-supervised acoustic model training has been well-explored in the context of hidden Markov Gaussian mixture models (HMM-GMMs), the re-emergence of deep neural network (DNN) acoustic models has given rise to some novel approaches to semi-supervised DNN training. This paper investigates several different strategies for semi-supervised DNN training, including the so-called ‘shared hidden layer’ approach and the ‘knowledge distillation’ (or student-teacher) approach. Particular attention is paid to the differing behaviour of semi-supervised DNN training methods during the cross-entropy and sequence training phases of model building. Experimental results on our internal study dataset provide evidence that in a low-resource scenario the most effective semi-supervised training strategy is ‘naive CE’ (treating manually transcribed and automatically transcribed data identically during the cross entropy phase of training) followed by use of a shared hidden layer technique during sequence training.