Natural-Logarithm-Rectified Activation Function in Convolutional Neural
  Networks
  Activation functions play a key role in providing remarkable performance in
deep neural networks, and the rectified linear unit (ReLU) is one of the most
widely used activation functions. Various new activation functions and
improvements on ReLU have been proposed, but each carry performance drawbacks.
In this paper, we propose an improved activation function, which we name the
natural-logarithm-rectified linear unit (NLReLU). This activation function uses
the parametric natural logarithmic transform to improve ReLU and is simply
defined as. NLReLU not only retains the sparse activation characteristic of
ReLU, but it also alleviates the "dying ReLU" and vanishing gradient problems
to some extent. It also reduces the bias shift effect and heteroscedasticity of
neuron data distributions among network layers in order to accelerate the
learning process. The proposed method was verified across ten convolutional
neural networks with different depths for two essential datasets. Experiments
illustrate that convolutional neural networks with NLReLU exhibit higher
accuracy than those with ReLU, and that NLReLU is comparable to other
well-known activation functions. NLReLU provides 0.16% and 2.04% higher
classification accuracy on average compared to ReLU when used in shallow
convolutional neural networks with the MNIST and CIFAR-10 datasets,
respectively. The average accuracy of deep convolutional neural networks with
NLReLU is 1.35% higher on average with the CIFAR-10 dataset.
