Robust Disentanglement of a Few Factors at a Time
  Disentanglement is at the forefront of unsupervised learning, as disentangled
representations of data improve generalization, interpretability, and
performance in downstream tasks. Current unsupervised approaches remain
inapplicable for real-world datasets since they are highly variable in their
performance and fail to reach levels of disentanglement of (semi-)supervised
approaches. We introduce population-based training (PBT) for improving
consistency in training variational autoencoders (VAEs) and demonstrate the
validity of this approach in a supervised setting (PBT-VAE). We then use
Unsupervised Disentanglement Ranking (UDR) as an unsupervised heuristic to
score models in our PBT-VAE training and show how models trained this way tend
to consistently disentangle only a subset of the generative factors. Building
on top of this observation we introduce the recursive rPU-VAE approach. We
train the model until convergence, remove the learned factors from the dataset
and reiterate. In doing so, we can label subsets of the dataset with the
learned factors and consecutively use these labels to train one model that
fully disentangles the whole dataset. With this approach, we show striking
improvement in state-of-the-art unsupervised disentanglement performance and
robustness across multiple datasets and metrics.
