A Survey on Self-supervised Pre-training for Sequential Transfer
  Learning in Neural Networks
  Deep neural networks are typically trained under a supervised learning
framework where a model learns a single task using labeled data. Instead of
relying solely on labeled data, practitioners can harness unlabeled or related
data to improve model performance, which is often more accessible and
ubiquitous. Self-supervised pre-training for transfer learning is becoming an
increasingly popular technique to improve state-of-the-art results using
unlabeled data. It involves first pre-training a model on a large amount of
unlabeled data, then adapting the model to target tasks of interest. In this
review, we survey self-supervised learning methods and their applications
within the sequential transfer learning framework. We provide an overview of
the taxonomy for self-supervised learning and transfer learning, and highlight
some prominent methods for designing pre-training tasks across different
domains. Finally, we discuss recent trends and suggest areas for future
investigation.
