Effects of the Nonlinearity in Activation Functions on the Performance
  of Deep Learning Models
  The nonlinearity of activation functions used in deep learning models are
crucial for the success of predictive models. There are several commonly used
simple nonlinear functions, including Rectified Linear Unit (ReLU) and
Leaky-ReLU (L-ReLU). In practice, these functions remarkably enhance the model
accuracy. However, there is limited insight into the functionality of these
nonlinear activation functions in terms of why certain models perform better
than others. Here, we investigate the model performance when using ReLU or
L-ReLU as activation functions in different model architectures and data
domains. Interestingly, we found that the application of L-ReLU is mostly
effective when the number of trainable parameters in a model is relatively
small. Furthermore, we found that the image classification models seem to
perform well with L-ReLU in fully connected layers, especially when pre-trained
models such as the VGG-16 are used for the transfer learning.
