Beyond Just Vision: A Review on Self-Supervised Representation Learning
  on Multimodal and Temporal Data
  Recently, Self-Supervised Representation Learning (SSRL) has attracted much
attention in the field of computer vision, speech, natural language processing
(NLP), and recently, with other types of modalities, including time series from
sensors. The popularity of self-supervised learning is driven by the fact that
traditional models typically require a huge amount of well-annotated data for
training. Acquiring annotated data can be a difficult and costly process.
Self-supervised methods have been introduced to improve the efficiency of
training data through discriminative pre-training of models using supervisory
signals that have been freely obtained from the raw data. Unlike existing
reviews of SSRL that have pre-dominately focused upon methods in the fields of
CV or NLP for a single modality, we aim to provide the first comprehensive
review of multimodal self-supervised learning methods for temporal data. To
this end, we 1) provide a comprehensive categorization of existing SSRL
methods, 2) introduce a generic pipeline by defining the key components of a
SSRL framework, 3) compare existing models in terms of their objective
function, network architecture and potential applications, and 4) review
existing multimodal techniques in each category and various modalities.
Finally, we present existing weaknesses and future opportunities. We believe
our work develops a perspective on the requirements of SSRL in domains that
utilise multimodal and/or temporal data
