Insights from Generative Modeling for Neural Video Compression
  While recent machine learning research has revealed connections between deep
generative models such as VAEs and rate-distortion losses used in learned
compression, most of this work has focused on images. In a similar spirit, we
view recently proposed neural video coding algorithms through the lens of deep
autoregressive and latent variable modeling. We present recent neural video
codecs as instances of a generalized stochastic temporal autoregressive
transform, and propose new avenues for further improvements inspired by
normalizing flows and structured priors. We propose several architectures that
yield state-of-the-art video compression performance on full-resolution video
and discuss their tradeoffs and ablations. In particular, we propose (i)
improved temporal autoregressive transforms, (ii) improved entropy models with
structured and temporal dependencies, and (iii) variable bitrate versions of
our algorithms. Since our improvements are compatible with a large class of
existing models, we provide further evidence that the generative modeling
viewpoint can advance the neural video coding field.
