ELF-VC: Efficient Learned Flexible-Rate Video Coding
  While learned video codecs have demonstrated great promise, they have yet to
achieve sufficient efficiency for practical deployment. In this work, we
propose several novel ideas for learned video compression which allow for
improved performance for the low-latency mode (I- and P-frames only) along with
a considerable increase in computational efficiency. In this setting, for
natural videos our approach compares favorably across the entire R-D curve
under metrics PSNR, MS-SSIM and VMAF against all mainstream video standards
(H.264, H.265, AV1) and all ML codecs. At the same time, our approach runs at
least 5x faster and has fewer parameters than all ML codecs which report these
figures.
  Our contributions include a flexible-rate framework allowing a single model
to cover a large and dense range of bitrates, at a negligible increase in
computation and parameter count; an efficient backbone optimized for ML-based
codecs; and a novel in-loop flow prediction scheme which leverages prior
information towards more efficient compression.
  We benchmark our method, which we call ELF-VC (Efficient, Learned and
Flexible Video Coding) on popular video test sets UVG and MCL-JCV under metrics
PSNR, MS-SSIM and VMAF. For example, on UVG under PSNR, it reduces the BD-rate
by 44% against H.264, 26% against H.265, 15% against AV1, and 35% against the
current best ML codec. At the same time, on an NVIDIA Titan V GPU our approach
encodes/decodes VGA at 49/91 FPS, HD 720 at 19/35 FPS, and HD 1080 at 10/18
FPS.
