Isometric Autoencoders
  High dimensional data is often assumed to be concentrated on or near a
low-dimensional manifold. Autoencoders (AE) is a popular technique to learn
representations of such data by pushing it through a neural network with a low
dimension bottleneck while minimizing a reconstruction error. Using high
capacity AE often leads to a large collection of minimizers, many of which
represent a low dimensional manifold that fits the data well but generalizes
poorly.
  Two sources of bad generalization are: extrinsic, where the learned manifold
possesses extraneous parts that are far from the data; and intrinsic, where the
encoder and decoder introduce arbitrary distortion in the low dimensional
parameterization. An approach taken to alleviate these issues is to add a
regularizer that favors a particular solution; common regularizers promote
sparsity, small derivatives, or robustness to noise.
  In this paper, we advocate an isometry (i.e., local distance preserving)
regularizer. Specifically, our regularizer encourages: (i) the decoder to be an
isometry; and (ii) the encoder to be the decoder's pseudo-inverse, that is, the
encoder extends the inverse of the decoder to the ambient space by orthogonal
projection. In a nutshell, (i) and (ii) fix both intrinsic and extrinsic
degrees of freedom and provide a non-linear generalization to principal
component analysis (PCA). Experimenting with the isometry regularizer on
dimensionality reduction tasks produces useful low-dimensional data
representations.
