Theory of gating in recurrent neural networks
  Recurrent neural networks (RNNs) are powerful dynamical models, widely used
in machine learning (ML) and neuroscience. Prior theoretical work has focused
on RNNs with additive interactions. However, gating - i.e. multiplicative -
interactions are ubiquitous in real neurons and also the central feature of the
best-performing RNNs in ML. Here, we show that gating offers flexible control
of two salient features of the collective dynamics: i) timescales and ii)
dimensionality. The gate controlling timescales leads to a novel, marginally
stable state, where the network functions as a flexible integrator. Unlike
previous approaches, gating permits this important function without parameter
fine-tuning or special symmetries. Gates also provide a flexible,
context-dependent mechanism to reset the memory trace, thus complementing the
memory function. The gate modulating the dimensionality can induce a novel,
discontinuous chaotic transition, where inputs push a stable system to strong
chaotic activity, in contrast to the typically stabilizing effect of inputs. At
this transition, unlike additive RNNs, the proliferation of critical points
(topological complexity) is decoupled from the appearance of chaotic dynamics
(dynamical complexity).
  The rich dynamics are summarized in phase diagrams, thus providing a map for
principled parameter initialization choices to ML practitioners.
