Similarity Analysis of Self-Supervised Speech Representations
  Self-supervised speech representation learning has recently been a prosperous
research topic. Many algorithms have been proposed for learning useful
representations from large-scale unlabeled data, and their applications to a
wide range of speech tasks have also been investigated. However, there has been
little research focusing on understanding the properties of existing
approaches. In this work, we aim to provide a comparative study of some of the
most representative self-supervised algorithms. Specifically, we quantify the
similarities between different self-supervised representations using existing
similarity measures. We also design probing tasks to study the correlation
between the models' pre-training loss and the amount of specific speech
information contained in their learned representations. In addition to showing
how various self-supervised models behave differently given the same input, our
study also finds that the training objective has a higher impact on
representation similarity than architectural choices such as building blocks
(RNN/Transformer/CNN) and directionality (uni/bidirectional). Our results also
suggest that there exists a strong correlation between pre-training loss and
downstream performance for some self-supervised algorithms.
