Symmetrical Gaussian Error Linear Units (SGELUs)
  In this paper, a novel neural network activation function, called Symmetrical
Gaussian Error Linear Unit (SGELU), is proposed to obtain high performance. It
is achieved by effectively integrating the property of the stochastic
regularizer in the Gaussian Error Linear Unit (GELU) with the symmetrical
characteristics. Combining with these two merits, the proposed unit introduces
the capability of the bidirection convergence to successfully optimize the
network without the gradient diminishing problem. The evaluations of SGELU
against GELU and Linearly Scaled Hyperbolic Tangent (LiSHT) have been carried
out on MNIST classification and MNIST auto-encoder, which provide great
validations in terms of the performance, the convergence rate among these
applications.
