Modeling the Nonsmoothness of Modern Neural Networks
  Modern neural networks have been successful in many regression-based tasks
such as face recognition, facial landmark detection, and image generation. In
this work, we investigate an intuitive but understudied characteristic of
modern neural networks, namely, the nonsmoothness. The experiments using
synthetic data confirm that such operations as ReLU and max pooling in modern
neural networks lead to nonsmoothness. We quantify the nonsmoothness using a
feature named the sum of the magnitude of peaks (SMP) and model the
input-output relationships for building blocks of modern neural networks.
Experimental results confirm that our model can accurately predict the
statistical behaviors of the nonsmoothness as it propagates through such
building blocks as the convolutional layer, the ReLU activation, and the max
pooling layer. We envision that the nonsmoothness feature can potentially be
used as a forensic tool for regression-based applications of neural networks.
