Deep inference of latent dynamics with spatio-temporal super-resolution
  using selective backpropagation through time
  Modern neural interfaces allow access to the activity of up to a million
neurons within brain circuits. However, bandwidth limits often create a
trade-off between greater spatial sampling (more channels or pixels) and the
temporal frequency of sampling. Here we demonstrate that it is possible to
obtain spatio-temporal super-resolution in neuronal time series by exploiting
relationships among neurons, embedded in latent low-dimensional population
dynamics. Our novel neural network training strategy, selective backpropagation
through time (SBTT), enables learning of deep generative models of latent
dynamics from data in which the set of observed variables changes at each time
step. The resulting models are able to infer activity for missing samples by
combining observations with learned latent dynamics. We test SBTT applied to
sequential autoencoders and demonstrate more efficient and higher-fidelity
characterization of neural population dynamics in electrophysiological and
calcium imaging data. In electrophysiology, SBTT enables accurate inference of
neuronal population dynamics with lower interface bandwidths, providing an
avenue to significant power savings for implanted neuroelectronic interfaces.
In applications to two-photon calcium imaging, SBTT accurately uncovers
high-frequency temporal structure underlying neural population activity,
substantially outperforming the current state-of-the-art. Finally, we
demonstrate that performance could be further improved by using limited,
high-bandwidth sampling to pretrain dynamics models, and then using SBTT to
adapt these models for sparsely-sampled data.
