Parametric Flatten-T Swish: An Adaptive Non-linear Activation Function
  For Deep Learning
  Activation function is a key component in deep learning that performs
non-linear mappings between the inputs and outputs. Rectified Linear Unit
(ReLU) has been the most popular activation function across the deep learning
community. However, ReLU contains several shortcomings that can result in
inefficient training of the deep neural networks, these are: 1) the negative
cancellation property of ReLU tends to treat negative inputs as unimportant
information for the learning, resulting in a performance degradation; 2) the
inherent predefined nature of ReLU is unlikely to promote additional
flexibility, expressivity, and robustness to the networks; 3) the mean
activation of ReLU is highly positive and leads to bias shift effect in network
layers; and 4) the multilinear structure of ReLU restricts the non-linear
approximation power of the networks. To tackle these shortcomings, this paper
introduced Parametric Flatten-T Swish (PFTS) as an alternative to ReLU. By
taking ReLU as a baseline method, the experiments showed that PFTS improved
classification accuracy on SVHN dataset by 0.31%, 0.98%, 2.16%, 17.72%, 1.35%,
0.97%, 39.99%, and 71.83% on DNN-3A, DNN-3B, DNN-4, DNN- 5A, DNN-5B, DNN-5C,
DNN-6, and DNN-7, respectively. Besides, PFTS also achieved the highest mean
rank among the comparison methods. The proposed PFTS manifested higher
non-linear approximation power during training and thereby improved the
predictive performance of the networks.
