A Mechanism for Producing Aligned Latent Spaces with Autoencoders
  Aligned latent spaces, where meaningful semantic shifts in the input space
correspond to a translation in the embedding space, play an important role in
the success of downstream tasks such as unsupervised clustering and data
imputation. In this work, we prove that linear and nonlinear autoencoders
produce aligned latent spaces by stretching along the left singular vectors of
the data. We fully characterize the amount of stretching in linear autoencoders
and provide an initialization scheme to arbitrarily stretch along the top
directions using these networks. We also quantify the amount of stretching in
nonlinear autoencoders in a simplified setting. We use our theoretical results
to align drug signatures across cell types in gene expression space and
semantic shifts in word embedding spaces.
