{'fraction_non_alphanumeric': 0.03697478991596639, 'fraction_numerical': 0.018487394957983194, 'mean_word_length': 4.838235294117647, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Sequence Distillation for Purely Sequence Trained Acoustic Models

This paper presents our exploration into teacher-student (TS) training for acoustic models (AMs) based on the lattice-free maximum mutual information technique. Whereas most previous studies of TS training used a frame-level distance between teacher and student models' distributions, we propose using the sequence-level temper-atured Kullback-Leibler divergence as a metric for TS training. In our experiment on the AMI meeting corpus, we prepared a strong teacher model consisting of a convolutional neural network, time delay neural network, and long short-term memory, which had 47.7M parameters and achieved a state-of-the-art word error rate (WER) of 18.05%. Whereas the small student AM (10.8M params. and 19.72% WER) trained by a frame-level TS training was able to fill only 43% of the WER gap between teacher and student AMs, the student AM trained by the proposed method achieved a 18.23% WER, filling 89% of the WER gap from the teacher AM. We also show that the frame-level TS training sometimes even degrades the performance of the student model whereas the proposed method consistently improved the accuracy.