Empirical Studies on the Properties of Linear Regions in Deep Neural
  Networks
  A deep neural network (DNN) with piecewise linear activations can partition
the input space into numerous small linear regions, where different linear
functions are fitted. It is believed that the number of these regions
represents the expressivity of the DNN. This paper provides a novel and
meticulous perspective to look into DNNs: Instead of just counting the number
of the linear regions, we study their local properties, such as the inspheres,
the directions of the corresponding hyperplanes, the decision boundaries, and
the relevance of the surrounding regions. We empirically observed that
different optimization techniques lead to completely different linear regions,
even though they result in similar classification accuracies. We hope our study
can inspire the design of novel optimization techniques, and help discover and
analyze the behaviors of DNNs.
