Transport Analysis of Infinitely Deep Neural Network
  We investigated the feature map inside deep neural networks (DNNs) by
tracking the transport map. We are interested in the role of depth (why do DNNs
perform better than shallow models?) and the interpretation of DNNs (what do
intermediate layers do?) Despite the rapid development in their application,
DNNs remain analytically unexplained because the hidden layers are nested and
the parameters are not faithful. Inspired by the integral representation of
shallow NNs, which is the continuum limit of the width, or the hidden unit
number, we developed the flow representation and transport analysis of DNNs.
The flow representation is the continuum limit of the depth or the hidden layer
number, and it is specified by an ordinary differential equation with a vector
field. We interpret an ordinary DNN as a transport map or a Euler broken line
approximation of the flow. Technically speaking, a dynamical system is a
natural model for the nested feature maps. In addition, it opens a new way to
the coordinate-free treatment of DNNs by avoiding the redundant parametrization
of DNNs. Following Wasserstein geometry, we analyze a flow in three aspects:
dynamical system, continuity equation, and Wasserstein gradient flow. A key
finding is that we specified a series of transport maps of the denoising
autoencoder (DAE). Starting from the shallow DAE, this paper develops three
topics: the transport map of the deep DAE, the equivalence between the stacked
DAE and the composition of DAEs, and the development of the double continuum
limit or the integral representation of the flow representation. As partial
answers to the research questions, we found that deeper DAEs converge faster
and the extracted features are better; in addition, a deep Gaussian DAE
transports mass to decrease the Shannon entropy of the data distribution.
