Improving Deep Learning by Inverse Square Root Linear Units (ISRLUs)
  We introduce the "inverse square root linear unit" (ISRLU) to speed up
learning in deep neural networks. ISRLU has better performance than ELU but has
many of the same benefits. ISRLU and ELU have similar curves and
characteristics. Both have negative values, allowing them to push mean unit
activation closer to zero, and bring the normal gradient closer to the unit
natural gradient, ensuring a noise-robust deactivation state, lessening the
over fitting risk. The significant performance advantage of ISRLU on
traditional CPUs also carry over to more efficient HW implementations on HW/SW
codesign for CNNs/RNNs. In experiments with TensorFlow, ISRLU leads to faster
learning and better generalization than ReLU on CNNs. This work also suggests a
computationally efficient variant called the "inverse square root unit" (ISRU)
which can be used for RNNs. Many RNNs use either long short-term memory (LSTM)
and gated recurrent units (GRU) which are implemented with tanh and sigmoid
activation functions. ISRU has less com- putational complexity but still has a
similar curve to tanh and sigmoid.
