S2RMs: Spatially Structured Recurrent Modules
  Capturing the structure of a data-generating process by means of appropriate
inductive biases can help in learning models that generalize well and are
robust to changes in the input distribution. While methods that harness spatial
and temporal structures find broad application, recent work has demonstrated
the potential of models that leverage sparse and modular structure using an
ensemble of sparingly interacting modules. In this work, we take a step towards
dynamic models that are capable of simultaneously exploiting both modular and
spatiotemporal structures. We accomplish this by abstracting the modeled
dynamical system as a collection of autonomous but sparsely interacting
sub-systems. The sub-systems interact according to a topology that is learned,
but also informed by the spatial structure of the underlying real-world system.
This results in a class of models that are well suited for modeling the
dynamics of systems that only offer local views into their state, along with
corresponding spatial locations of those views. On the tasks of video
prediction from cropped frames and multi-agent world modeling from partial
observations in the challenging Starcraft2 domain, we find our models to be
more robust to the number of available views and better capable of
generalization to novel tasks without additional training, even when compared
against strong baselines that perform equally well or better on the training
distribution.
