Understanding Latent Correlation-Based Multiview Learning and
  Self-Supervision: An Identifiability Perspective
  Multiple views of data, both naturally acquired (e.g., image and audio) and
artificially produced (e.g., via adding different noise to data samples), have
proven useful in enhancing representation learning. Natural views are often
handled by multiview analysis tools, e.g., (deep) canonical correlation
analysis [(D)CCA], while the artificial ones are frequently used in
self-supervised learning (SSL) paradigms, e.g., BYOL and Barlow Twins. Both
types of approaches often involve learning neural feature extractors such that
the embeddings of data exhibit high cross-view correlations. Although
intuitive, the effectiveness of correlation-based neural embedding is mostly
empirically validated.
  This work aims to understand latent correlation maximization-based deep
multiview learning from a latent component identification viewpoint. An
intuitive generative model of multiview data is adopted, where the views are
different nonlinear mixtures of shared and private components. Since the shared
components are view/distortion-invariant, representing the data using such
components is believed to reveal the identity of the samples effectively and
robustly. Under this model, latent correlation maximization is shown to
guarantee the extraction of the shared components across views (up to certain
ambiguities). In addition, it is further shown that the private information in
each view can be provably disentangled from the shared using proper
regularization design. A finite sample analysis, which has been rare in
nonlinear mixture identifiability study, is also presented. The theoretical
results and newly designed regularization are tested on a series of tasks.
