Disentangling by Partitioning: A Representation Learning Framework for
  Multimodal Sensory Data
  Multimodal sensory data resembles the form of information perceived by humans
for learning, and are easy to obtain in large quantities. Compared to unimodal
data, synchronization of concepts between modalities in such data provides
supervision for disentangling the underlying explanatory factors of each
modality. Previous work leveraging multimodal data has mainly focused on
retaining only the modality-invariant factors while discarding the rest. In
this paper, we present a partitioned variational autoencoder (PVAE) and several
training objectives to learn disentangled representations, which encode not
only the shared factors, but also modality-dependent ones, into separate latent
variables. Specifically, PVAE integrates a variational inference framework and
a multimodal generative model that partitions the explanatory factors and
conditions only on the relevant subset of them for generation. We evaluate our
model on two parallel speech/image datasets, and demonstrate its ability to
learn disentangled representations by qualitatively exploring within-modality
and cross-modality conditional generation with semantics and styles specified
by examples. For quantitative analysis, we evaluate the classification accuracy
of automatically discovered semantic units. Our PVAE can achieve over 99%
accuracy on both modalities.
