Activation Functions: Dive into an optimal activation function
  Activation functions have come up as one of the essential components of
neural networks. The choice of adequate activation function can impact the
accuracy of these methods. In this study, we experiment for finding an optimal
activation function by defining it as a weighted sum of existing activation
functions and then further optimizing these weights while training the network.
The study uses three activation functions, ReLU, tanh, and sin, over three
popular image datasets, MNIST, FashionMNIST, and KMNIST. We observe that the
ReLU activation function can easily overlook other activation functions. Also,
we see that initial layers prefer to have ReLU or LeakyReLU type of activation
functions, but deeper layers tend to prefer more convergent activation
functions.
