Data Overlap: A Prerequisite For Disentanglement
  Learning disentangled representations with variational autoencoders (VAEs) is
often attributed to the regularisation component of the loss. In this work, we
highlight the interaction between data and the reconstruction term of the loss
as the main contributor to disentanglement in VAEs. We note that standardised
benchmark datasets are constructed in a way that is conducive to learning what
appear to be disentangled representations. We design an intuitive adversarial
dataset that exploits this mechanism to break existing state-of-the-art
disentanglement frameworks. Finally, we provide solutions in the form of a
modified reconstruction loss suggesting that VAEs are accidental distance
learners.
