Parametric Exponential Linear Unit for Deep Convolutional Neural
  Networks
  Object recognition is an important task for improving the ability of visual
systems to perform complex scene understanding. Recently, the Exponential
Linear Unit (ELU) has been proposed as a key component for managing bias shift
in Convolutional Neural Networks (CNNs), but defines a parameter that must be
set by hand. In this paper, we propose learning a parameterization of ELU in
order to learn the proper activation shape at each layer in the CNNs. Our
results on the MNIST, CIFAR-10/100 and ImageNet datasets using the NiN,
Overfeat, All-CNN and ResNet networks indicate that our proposed Parametric ELU
(PELU) has better performances than the non-parametric ELU. We have observed as
much as a 7.28% relative error improvement on ImageNet with the NiN network,
with only 0.0003% parameter increase. Our visual examination of the non-linear
behaviors adopted by Vgg using PELU shows that the network took advantage of
the added flexibility by learning different activations at different layers.
