Measuring Model Complexity of Neural Networks with Curve Activation
  Functions
  It is fundamental to measure model complexity of deep neural networks. The
existing literature on model complexity mainly focuses on neural networks with
piecewise linear activation functions. Model complexity of neural networks with
general curve activation functions remains an open problem. To tackle the
challenge, in this paper, we first propose the linear approximation neural
network (LANN for short), a piecewise linear framework to approximate a given
deep model with curve activation function. LANN constructs individual piecewise
linear approximation for the activation function of each neuron, and minimizes
the number of linear regions to satisfy a required approximation degree. Then,
we analyze the upper bound of the number of linear regions formed by LANNs, and
derive the complexity measure based on the upper bound. To examine the
usefulness of the complexity measure, we experimentally explore the training
process of neural networks and detect overfitting. Our results demonstrate that
the occurrence of overfitting is positively correlated with the increase of
model complexity during training. We find that the $L^1$ and $L^2$
regularizations suppress the increase of model complexity. Finally, we propose
two approaches to prevent overfitting by directly constraining model
complexity, namely neuron pruning and customized $L^1$ regularization.
