A Linear Systems Theory of Normalizing Flows
  Normalizing Flows are a promising new class of algorithms for unsupervised
learning based on maximum likelihood optimization with change of variables.
They offer to learn a factorized component representation for complex nonlinear
data and, simultaneously, yield a density function that can evaluate
likelihoods and generate samples. Despite these diverse offerings, applications
of Normalizing Flows have focused primarily on sampling and likelihoods, with
little emphasis placed on feature representation. A lack of theoretical
foundation has left many open questions about how to interpret and apply the
learned components of the model. We provide a new theoretical perspective of
Normalizing Flows using the lens of linear systems theory, showing that optimal
flows learn to represent the local covariance at each region of input space.
Using this insight, we develop a new algorithm to extract interpretable
component representations from the learned model, where components correspond
to Cartesian dimensions and are scaled according to their manifold
significance. In addition, we highlight a stability concern for the learning
algorithm that was previously unaddressed, providing a theoretically-grounded
solution to mediate the problem. Experiments with toy manifold learning
datasets, as well as the MNIST image dataset, provide convincing support for
our theory and tools.
