Instance-Adaptive Video Compression: Improving Neural Codecs by Training
  on the Test Set
  We introduce a video compression algorithm based on instance-adaptive
learning. On each video sequence to be transmitted, we finetune a pretrained
compression model. The optimal parameters are transmitted to the receiver along
with the latent code. By entropy-coding the parameter updates under a suitable
mixture model prior, we ensure that the network parameters can be encoded
efficiently. This instance-adaptive compression algorithm is agnostic about the
choice of base model and has the potential to improve any neural video codec.
On UVG, HEVC, and Xiph datasets, our codec improves the performance of a
low-latency scale-space flow model by between 21% and 26% BD-rate savings, and
that of a state-of-the-art B-frame model by 17 to 20% BD-rate savings. We also
demonstrate that instance-adaptive finetuning improves the robustness to domain
shift. Finally, our approach reduces the capacity requirements on compression
models. We show that it enables a state-of-the-art performance even after
reducing the network size by 72%.
