SlimIPL: Language-Model-Free Iterative Pseudo-Labeling
  Recent results in end-to-end automatic speech recognition have demonstrated
the efficacy of pseudo-labeling for semi-supervised models trained both with
Connectionist Temporal Classification (CTC) and Sequence-to-Sequence (seq2seq)
losses. Iterative Pseudo-Labeling (IPL), which continuously trains a single
model using pseudo-labels iteratively re-generated as the model learns, has
been shown to further improve performance in ASR. We improve upon the IPL
algorithm: as the model learns, we propose to iteratively re-generate
transcriptions with hard labels (the most probable tokens), that is, without a
language model. We call this approach Language-Model-Free IPL (slimIPL) and
give a resultant training setup for low-resource settings with CTC-based
models. slimIPL features a dynamic cache for pseudo-labels which reduces
sensitivity to changes in relabeling hyperparameters and results in improves
training stability. slimIPL is also highly-efficient and requires 3.5-4x fewer
computational resources to converge than other state-of-the-art
semi/self-supervised approaches. With only 10 hours of labeled audio, slimIPL
is competitive with self-supervised approaches, and is state-of-the-art with
100 hours of labeled audio without the use of a language model both at test
time and during pseudo-label generation.
