Deep Multi-View Learning via Task-Optimal CCA
  Canonical Correlation Analysis (CCA) is widely used for multimodal data
analysis and, more recently, for discriminative tasks such as multi-view
learning; however, it makes no use of class labels. Recent CCA methods have
started to address this weakness but are limited in that they do not
simultaneously optimize the CCA projection for discrimination and the CCA
projection itself, or they are linear only. We address these deficiencies by
simultaneously optimizing a CCA-based and a task objective in an end-to-end
manner. Together, these two objectives learn a non-linear CCA projection to a
shared latent space that is highly correlated and discriminative. Our method
shows a significant improvement over previous state-of-the-art (including deep
supervised approaches) for cross-view classification, regularization with a
second view, and semi-supervised learning on real data.
