RNNs of RNNs: Recursive Construction of Stable Assemblies of Recurrent
  Neural Networks
  Recurrent neural networks (RNNs) are widely used throughout neuroscience as
models of local neural activity. Many properties of single RNNs are well
characterized theoretically, but experimental neuroscience has moved in the
direction of studying multiple interacting areas, and RNN theory needs to be
likewise extended. We take a constructive approach towards this problem,
leveraging tools from nonlinear control theory and machine learning to
characterize when combinations of stable RNNs will themselves be stable.
Importantly, we derive conditions which allow for massive feedback connections
between interacting RNNs. We parameterize these conditions for easy
optimization using gradient-based techniques, and show that
stability-constrained `network of networks' can perform well on challenging
sequential-processing benchmark tasks. Altogether, our results provide a
principled approach towards understanding distributed, modular function in the
brain.
