MoViNets: Mobile Video Networks for Efficient Video Recognition
  We present Mobile Video Networks (MoViNets), a family of computation and
memory efficient video networks that can operate on streaming video for online
inference. 3D convolutional neural networks (CNNs) are accurate at video
recognition but require large computation and memory budgets and do not support
online inference, making them difficult to work on mobile devices. We propose a
three-step approach to improve computational efficiency while substantially
reducing the peak memory usage of 3D CNNs. First, we design a video network
search space and employ neural architecture search to generate efficient and
diverse 3D CNN architectures. Second, we introduce the Stream Buffer technique
that decouples memory from video clip duration, allowing 3D CNNs to embed
arbitrary-length streaming video sequences for both training and inference with
a small constant memory footprint. Third, we propose a simple ensembling
technique to improve accuracy further without sacrificing efficiency. These
three progressive techniques allow MoViNets to achieve state-of-the-art
accuracy and efficiency on the Kinetics, Moments in Time, and Charades video
action recognition datasets. For instance, MoViNet-A5-Stream achieves the same
accuracy as X3D-XL on Kinetics 600 while requiring 80% fewer FLOPs and 65% less
memory. Code will be made available at
https://github.com/tensorflow/models/tree/master/official/vision.
