How Does Pre-trained Wav2Vec2.0 Perform on Domain Shifted ASR? An
  Extensive Benchmark on Air Traffic Control Communications
  Recent work on self-supervised pre-training focus on leveraging large-scale
unlabeled speech data to build robust end-to-end (E2E) acoustic models (AM)
that can be later fine-tuned on downstream tasks e.g., automatic speech
recognition (ASR). Yet, few works investigated the impact on performance when
the data substantially differs between the pre-training and downstream
fine-tuning phases (i.e., domain shift). We target this scenario by analyzing
the robustness of Wav2Vec2.0 and XLS-R models on downstream ASR for a
completely unseen domain, i.e., air traffic control (ATC) communications. We
benchmark the proposed models on four challenging ATC test sets
(signal-to-noise ratio varies between 5 to 20 dB). Relative word error rate
(WER) reduction between 20% to 40% are obtained in comparison to hybrid-based
state-of-the-art ASR baselines by fine-tuning E2E acoustic models with a small
fraction of labeled data. We also study the impact of fine-tuning data size on
WERs, going from 5 minutes (few-shot) to 15 hours.
