Conditional independence for pretext task selection in Self-supervised
  speech representation learning
  Through solving pretext tasks, self-supervised learning (SSL) leverages
unlabeled data to extract useful latent representations replacing traditional
input features in the downstream task. A common pretext task consists in
pretraining a SSL model on pseudo-labels derived from the original signal. This
technique is particularly relevant for speech data where various meaningful
signal processing features may serve as pseudo-labels. However, the process of
selecting pseudo-labels, for speech or other types of data, remains mostly
unexplored and currently relies on observing the results on the final
downstream task. Nevertheless, this methodology is not sustainable at scale due
to substantial computational (hence carbon) costs. Thus, this paper introduces
a practical and theoretical framework to select relevant pseudo-labels with
respect to a given downstream task. More precisely, we propose a functional
estimator of the pseudo-label utility grounded in the conditional independence
theory, which does not require any training. The experiments conducted on
speaker recognition and automatic speech recognition validate our estimator,
showing a significant correlation between the performance observed on the
downstream task and the utility estimates obtained with our approach,
facilitating the prospection of relevant pseudo-labels for self-supervised
speech representation learning.
