Recurrent Neural Networks with Flexible Gates using Kernel Activation
  Functions
  Gated recurrent neural networks have achieved remarkable results in the
analysis of sequential data. Inside these networks, gates are used to control
the flow of information, allowing to model even very long-term dependencies in
the data. In this paper, we investigate whether the original gate equation (a
linear projection followed by an element-wise sigmoid) can be improved. In
particular, we design a more flexible architecture, with a small number of
adaptable parameters, which is able to model a wider range of gating functions
than the classical one. To this end, we replace the sigmoid function in the
standard gate with a non-parametric formulation extending the recently proposed
kernel activation function (KAF), with the addition of a residual
skip-connection. A set of experiments on sequential variants of the MNIST
dataset shows that the adoption of this novel gate allows to improve accuracy
with a negligible cost in terms of computational power and with a large
speed-up in the number of training iterations.
