Multi-view Alignment and Generation in CCA via Consistent Latent
  Encoding
  Multi-view alignment, achieving one-to-one correspondence of multi-view
inputs, is critical in many real-world multi-view applications, especially for
cross-view data analysis problems. Recently, an increasing number of works
study this alignment problem with Canonical Correlation Analysis (CCA).
However, existing CCA models are prone to misalign the multiple views due to
either the neglect of uncertainty or the inconsistent encoding of the multiple
views. To tackle these two issues, this paper studies multi-view alignment from
the Bayesian perspective. Delving into the impairments of inconsistent
encodings, we propose to recover correspondence of the multi-view inputs by
matching the marginalization of the joint distribution of multi-view random
variables under different forms of factorization. To realize our design, we
present Adversarial CCA (ACCA) which achieves consistent latent encodings by
matching the marginalized latent encodings through the adversarial training
paradigm. Our analysis based on conditional mutual information reveals that
ACCA is flexible for handling implicit distributions. Extensive experiments on
correlation analysis and cross-view generation under noisy input settings
demonstrate the superiority of our model.
