Multimodal Representation Learning using Deep Multiset Canonical
  Correlation
  We propose Deep Multiset Canonical Correlation Analysis (dMCCA) as an
extension to representation learning using CCA when the underlying signal is
observed across multiple (more than two) modalities. We use deep learning
framework to learn non-linear transformations from different modalities to a
shared subspace such that the representations maximize the ratio of between-
and within-modality covariance of the observations. Unlike linear discriminant
analysis, we do not need class information to learn these representations, and
we show that this model can be trained for complex data using mini-batches.
Using synthetic data experiments, we show that dMCCA can effectively recover
the common signal across the different modalities corrupted by multiplicative
and additive noise. We also analyze the sensitivity of our model to recover the
correlated components with respect to mini-batch size and dimension of the
embeddings. Performance evaluation on noisy handwritten datasets shows that our
model outperforms other CCA-based approaches and is comparable to deep neural
network models trained end-to-end on this dataset.
