The Quest for the Golden Activation Function
  Deep Neural Networks have been shown to be beneficial for a variety of tasks,
in particular allowing for end-to-end learning and reducing the requirement for
manual design decisions. However, still many parameters have to be chosen in
advance, also raising the need to optimize them. One important, but often
ignored system parameter is the selection of a proper activation function.
Thus, in this paper we target to demonstrate the importance of activation
functions in general and show that for different tasks different activation
functions might be meaningful. To avoid the manual design or selection of
activation functions, we build on the idea of genetic algorithms to learn the
best activation function for a given task. In addition, we introduce two new
activation functions, ELiSH and HardELiSH, which can easily be incorporated in
our framework. In this way, we demonstrate for three different image
classification benchmarks that different activation functions are learned, also
showing improved results compared to typically used baselines.
