Understanding and Mitigating Exploding Inverses in Invertible Neural
  Networks
  Invertible neural networks (INNs) have been used to design generative models,
implement memory-saving gradient computation, and solve inverse problems. In
this work, we show that commonly-used INN architectures suffer from exploding
inverses and are thus prone to becoming numerically non-invertible. Across a
wide range of INN use-cases, we reveal failures including the non-applicability
of the change-of-variables formula on in- and out-of-distribution (OOD) data,
incorrect gradients for memory-saving backprop, and the inability to sample
from normalizing flow models. We further derive bi-Lipschitz properties of
atomic building blocks of common architectures. These insights into the
stability of INNs then provide ways forward to remedy these failures. For tasks
where local invertibility is sufficient, like memory-saving backprop, we
propose a flexible and efficient regularizer. For problems where global
invertibility is necessary, such as applying normalizing flows on OOD data, we
show the importance of designing stable INN building blocks.
