Independent Subspace Analysis for Unsupervised Learning of Disentangled
  Representations
  Recently there has been an increased interest in unsupervised learning of
disentangled representations using the Variational Autoencoder (VAE) framework.
Most of the existing work has focused largely on modifying the variational cost
function to achieve this goal. We first show that these modifications, e.g.
beta-VAE, simplify the tendency of variational inference to underfit causing
pathological over-pruning and over-orthogonalization of learned components.
Second we propose a complementary approach: to modify the probabilistic model
with a structured latent prior. This prior allows to discover latent variable
representations that are structured into a hierarchy of independent vector
spaces. The proposed prior has three major advantages: First, in contrast to
the standard VAE normal prior the proposed prior is not rotationally invariant.
This resolves the problem of unidentifiability of the standard VAE normal
prior. Second, we demonstrate that the proposed prior encourages a disentangled
latent representation which facilitates learning of disentangled
representations. Third, extensive quantitative experiments demonstrate that the
prior significantly mitigates the trade-off between reconstruction loss and
disentanglement over the state of the art.
