Oscillatory Fourier Neural Network: A Compact and Efficient Architecture
  for Sequential Processing
  Tremendous progress has been made in sequential processing with the recent
advances in recurrent neural networks. However, recurrent architectures face
the challenge of exploding/vanishing gradients during training, and require
significant computational resources to execute back-propagation through time.
Moreover, large models are typically needed for executing complex sequential
tasks. To address these challenges, we propose a novel neuron model that has
cosine activation with a time varying component for sequential processing. The
proposed neuron provides an efficient building block for projecting sequential
inputs into spectral domain, which helps to retain long-term dependencies with
minimal extra model parameters and computation. A new type of recurrent network
architecture, named Oscillatory Fourier Neural Network, based on the proposed
neuron is presented and applied to various types of sequential tasks. We
demonstrate that recurrent neural network with the proposed neuron model is
mathematically equivalent to a simplified form of discrete Fourier transform
applied onto periodical activation. In particular, the computationally
intensive back-propagation through time in training is eliminated, leading to
faster training while achieving the state of the art inference accuracy in a
diverse group of sequential tasks. For instance, applying the proposed model to
sentiment analysis on IMDB review dataset reaches 89.4% test accuracy within 5
epochs, accompanied by over 35x reduction in the model size compared to LSTM.
The proposed novel RNN architecture is well poised for intelligent sequential
processing in resource constrained hardware.
