A survey on recently proposed activation functions for Deep Learning
  Artificial neural networks (ANN), typically referred to as neural networks,
are a class of Machine Learning algorithms and have achieved widespread
success, having been inspired by the biological structure of the human brain.
Neural networks are inherently powerful due to their ability to learn complex
function approximations from data. This generalization ability has been able to
impact multidisciplinary areas involving image recognition, speech recognition,
natural language processing, and others. Activation functions are a crucial
sub-component of neural networks. They define the output of a node in the
network given a set of inputs. This survey discusses the main concepts of
activation functions in neural networks, including; a brief introduction to
deep neural networks, a summary of what are activation functions and how they
are used in neural networks, their most common properties, the different types
of activation functions, some of the challenges, limitations, and alternative
solutions faced by activation functions, concluding with the final remarks.
