Self-Supervised Audio-and-Text Pre-training with Extremely Low-Resource
  Parallel Data
  Multimodal pre-training for audio-and-text has recently been proved to be
effective and has significantly improved the performance of many downstream
speech understanding tasks. However, these state-of-the-art pre-training
audio-text models work well only when provided with large amount of parallel
audio-and-text data, which brings challenges on many languages that are rich in
unimodal corpora but scarce of parallel cross-modal corpus. In this paper, we
investigate whether it is possible to pre-train an audio-text multimodal model
with extremely low-resource parallel data and extra non-parallel unimodal data.
Our pre-training framework consists of the following components: (1)
Intra-modal Denoising Auto-Encoding (IDAE), which is able to reconstruct input
text (audio) representations from a noisy version of itself. (2) Cross-modal
Denoising Auto-Encoding (CDAE), which is pre-trained to reconstruct the input
text (audio), given both a noisy version of the input text (audio) and the
corresponding translated noisy audio features (text embeddings). (3) Iterative
Denoising Process (IDP), which iteratively translates raw audio (text) and the
corresponding text embeddings (audio features) translated from previous
iteration into the new less-noisy text embeddings (audio features). We adapt a
dual cross-modal Transformer as our backbone model which consists of two
unimodal encoders for IDAE and two cross-modal encoders for CDAE and IDP. Our
method achieves comparable performance on multiple downstream speech
understanding tasks compared with the model pre-trained on fully parallel data,
demonstrating the great potential of the proposed method. Our code is available
at: \url{https://github.com/KarlYuKang/Low-Resource-Multimodal-Pre-training}.
