Invertible Residual Networks
  We show that standard ResNet architectures can be made invertible, allowing
the same model to be used for classification, density estimation, and
generation. Typically, enforcing invertibility requires partitioning dimensions
or restricting network architectures. In contrast, our approach only requires
adding a simple normalization step during training, already available in
standard frameworks. Invertible ResNets define a generative model which can be
trained by maximum likelihood on unlabeled data. To compute likelihoods, we
introduce a tractable approximation to the Jacobian log-determinant of a
residual block. Our empirical evaluation shows that invertible ResNets perform
competitively with both state-of-the-art image classifiers and flow-based
generative models, something that has not been previously achieved with a
single architecture.
