Warming-up recurrent neural networks to maximize reachable
  multi-stability greatly improves learning
  Training recurrent neural networks is known to be difficult when time
dependencies become long. Consequently, training standard gated cells such as
gated recurrent units and long-short term memory on benchmarks where long-term
memory is required remains an arduous task. In this work, we propose a general
way to initialize any recurrent network connectivity through a process called
"warm-up" to improve its capability to learn arbitrarily long time
dependencies. This initialization process is designed to maximize network
reachable multi-stability, i.e. the number of attractors within the network
that can be reached through relevant input trajectories. Warming-up is
performed before training, using stochastic gradient descent on a specifically
designed loss. We show that warming-up greatly improves recurrent neural
network performance on long-term memory benchmarks for multiple recurrent cell
types, but can sometimes impede precision. We therefore introduce a parallel
recurrent network structure with partial warm-up that is shown to greatly
improve learning on long time-series while maintaining high levels of
precision. This approach provides a general framework for improving learning
abilities of any recurrent cell type when long-term memory is required.
