A Survey on Activation Functions and their relation with Xavier and He
  Normal Initialization
  In artificial neural network, the activation function and the weight
initialization method play important roles in training and performance of a
neural network. The question arises is what properties of a function are
important/necessary for being a well-performing activation function. Also, the
most widely used weight initialization methods - Xavier and He normal
initialization have fundamental connection with activation function. This
survey discusses the important/necessary properties of activation function and
the most widely used activation functions (sigmoid, tanh, ReLU, LReLU and
PReLU). This survey also explores the relationship between these activation
functions and the two weight initialization methods - Xavier and He normal
initialization.
