End-to-End Learning for Video Frame Compression with Self-Attention
  One of the core components of conventional (i.e., non-learned) video codecs
consists of predicting a frame from a previously-decoded frame, by leveraging
temporal correlations. In this paper, we propose an end-to-end learned system
for compressing video frames. Instead of relying on pixel-space motion (as with
optical flow), our system learns deep embeddings of frames and encodes their
difference in latent space. At decoder-side, an attention mechanism is designed
to attend to the latent space of frames to decide how different parts of the
previous and current frame are combined to form the final predicted current
frame. Spatially-varying channel allocation is achieved by using importance
masks acting on the feature-channels. The model is trained to reduce the
bitrate by minimizing a loss on importance maps and a loss on the probability
output by a context model for arithmetic coding. In our experiments, we show
that the proposed system achieves high compression rates and high objective
visual quality as measured by MS-SSIM and PSNR. Furthermore, we provide
ablation studies where we highlight the contribution of different components.
