Population-based Global Optimisation Methods for Learning Long-term
  Dependencies with RNNs
  Despite recent innovations in network architectures and loss functions,
training RNNs to learn long-term dependencies remains difficult due to
challenges with gradient-based optimisation methods. Inspired by the success of
Deep Neuroevolution in reinforcement learning (Such et al. 2017), we explore
the use of gradient-free population-based global optimisation (PBO) techniques
-- training RNNs to capture long-term dependencies in time-series data. Testing
evolution strategies (ES) and particle swarm optimisation (PSO) on an
application in volatility forecasting, we demonstrate that PBO methods lead to
performance improvements in general, with ES exhibiting the most consistent
results across a variety of architectures.
