Theoretical Exploration of Solutions of Feedforward ReLU networks
  This paper aims to interpret the mechanism of feedforward ReLU networks by
exploring their solutions for piecewise linear functions through basic rules.
The constructed solutions should be universal enough to explain the network
architectures of engineering. In order for that, we borrow the methodology of
theoretical physics to develop the theories. Some of the consequences of our
theories include: Under geometric backgrounds, the solutions of both
three-layer networks and deep-layer networks are presented, and the solution
universality is ensured by several ways; We give clear and intuitive
interpretations of each component of network architectures, such as the
parameter-sharing mechanism for multi-output, the function of each layer, the
advantage of deep layers, the redundancy of parameters, and so on. We explain
three typical network architectures: the subnetwork of last three layers of
convolutional networks, multi-layer feedforward networks, and the decoder of
autoencoders. This paper is expected to provide a basic foundation of theories
of feedforward ReLU networks for further investigations.
