Flatten-T Swish: a thresholded ReLU-Swish-like activation function for
  deep learning
  Activation functions are essential for deep learning methods to learn and
perform complex tasks such as image classification. Rectified Linear Unit
(ReLU) has been widely used and become the default activation function across
the deep learning community since 2012. Although ReLU has been popular,
however, the hard zero property of the ReLU has heavily hindered the negative
values from propagating through the network. Consequently, the deep neural
network has not been benefited from the negative representations. In this work,
an activation function called Flatten-T Swish (FTS) that leverage the benefit
of the negative values is proposed. To verify its performance, this study
evaluates FTS with ReLU and several recent activation functions. Each
activation function is trained using MNIST dataset on five different deep fully
connected neural networks (DFNNs) with depth vary from five to eight layers.
For a fair evaluation, all DFNNs are using the same configuration settings.
Based on the experimental results, FTS with a threshold value, T=-0.20 has the
best overall performance. As compared with ReLU, FTS (T=-0.20) improves MNIST
classification accuracy by 0.13%, 0.70%, 0.67%, 1.07% and 1.15% on wider 5
layers, slimmer 5 layers, 6 layers, 7 layers and 8 layers DFNNs respectively.
Apart from this, the study also noticed that FTS converges twice as fast as
ReLU. Although there are other existing activation functions are also
evaluated, this study elects ReLU as the baseline activation function.
