An Analysis of State-of-the-art Activation Functions For Supervised Deep
  Neural Network
  This paper provides an analysis of state-of-the-art activation functions with
respect to supervised classification of deep neural network. These activation
functions comprise of Rectified Linear Units (ReLU), Exponential Linear Unit
(ELU), Scaled Exponential Linear Unit (SELU), Gaussian Error Linear Unit
(GELU), and the Inverse Square Root Linear Unit (ISRLU). To evaluate,
experiments over two deep learning network architectures integrating these
activation functions are conducted. The first model, basing on Multilayer
Perceptron (MLP), is evaluated with MNIST dataset to perform these activation
functions. Meanwhile, the second model, likely VGGish-based architecture, is
applied for Acoustic Scene Classification (ASC) Task 1A in DCASE 2018
challenge, thus evaluate whether these activation functions work well in
different datasets as well as different network architectures.
