Learn-able parameter guided Activation Functions
  In this paper, we explore the concept of adding learn-able slope and mean
shift parameters to an activation function to improve the total response
region. The characteristics of an activation function depend highly on the
value of parameters. Making the parameters learn-able, makes the activation
function more dynamic and capable to adapt as per the requirements of its
neighboring layers. The introduced slope parameter is independent of other
parameters in the activation function. The concept was applied to ReLU to
develop Dual Line and DualParametric ReLU activation function. Evaluation on
MNIST and CIFAR10 show that the proposed activation function Dual Line achieves
top-5 position for mean accuracy among 43 activation functions tested with
LENET4, LENET5, and WideResNet architectures. This is the first time more than
40 activation functions were analyzed on MNIST andCIFAR10 dataset at the same
time. The study on the distribution of positive slope parameter beta indicates
that the activation function adapts as per the requirements of the neighboring
layers. The study shows that model performance increases with the proposed
activation functions
