A Methodology for Automatic Selection of Activation Functions to Design
  Hybrid Deep Neural Networks
  Activation functions influence behavior and performance of DNNs. Nonlinear
activation functions, like Rectified Linear Units (ReLU), Exponential Linear
Units (ELU) and Scaled Exponential Linear Units (SELU), outperform the linear
counterparts. However, selecting an appropriate activation function is a
challenging problem, as it affects the accuracy and the complexity of the given
DNN. In this paper, we propose a novel methodology to automatically select the
best-possible activation function for each layer of a given DNN, such that the
overall DNN accuracy, compared to considering only one type of activation
function for the whole DNN, is improved. However, an associated scientific
challenge in exploring all the different configurations of activation functions
would be time and resource-consuming. Towards this, our methodology identifies
the Evaluation Points during learning to evaluate the accuracy in an
intermediate step of training and to perform early termination by checking the
accuracy gradient of the learning curve. This helps in significantly reducing
the exploration time during training. Moreover, our methodology selects, for
each layer, the dropout rate that optimizes the accuracy. Experiments show that
we are able to achieve on average 7% to 15% Relative Error Reduction on MNIST,
CIFAR-10 and CIFAR-100 benchmarks, with limited performance and power penalty
on GPUs.
