Learning activation functions from data using cubic spline interpolation
  Neural networks require a careful design in order to perform properly on a
given task. In particular, selecting a good activation function (possibly in a
data-dependent fashion) is a crucial step, which remains an open problem in the
research community. Despite a large amount of investigations, most current
implementations simply select one fixed function from a small set of
candidates, which is not adapted during training, and is shared among all
neurons throughout the different layers. However, neither two of these
assumptions can be supposed optimal in practice. In this paper, we present a
principled way to have data-dependent adaptation of the activation functions,
which is performed independently for each neuron. This is achieved by
leveraging over past and present advances on cubic spline interpolation,
allowing for local adaptation of the functions around their regions of use. The
resulting algorithm is relatively cheap to implement, and overfitting is
counterbalanced by the inclusion of a novel damping criterion, which penalizes
unwanted oscillations from a predefined shape. Experimental results validate
the proposal over two well-known benchmarks.
