Spatiotemporal Contrastive Video Representation Learning
  We present a self-supervised Contrastive Video Representation Learning (CVRL)
method to learn spatiotemporal visual representations from unlabeled videos.
Our representations are learned using a contrastive loss, where two augmented
clips from the same short video are pulled together in the embedding space,
while clips from different videos are pushed away. We study what makes for good
data augmentations for video self-supervised learning and find that both
spatial and temporal information are crucial. We carefully design data
augmentations involving spatial and temporal cues. Concretely, we propose a
temporally consistent spatial augmentation method to impose strong spatial
augmentations on each frame of the video while maintaining the temporal
consistency across frames. We also propose a sampling-based temporal
augmentation method to avoid overly enforcing invariance on clips that are
distant in time. On Kinetics-600, a linear classifier trained on the
representations learned by CVRL achieves 70.4% top-1 accuracy with a
3D-ResNet-50 (R3D-50) backbone, outperforming ImageNet supervised pre-training
by 15.7% and SimCLR unsupervised pre-training by 18.8% using the same inflated
R3D-50. The performance of CVRL can be further improved to 72.9% with a larger
R3D-152 (2x filters) backbone, significantly closing the gap between
unsupervised and supervised video representation learning. Our code and models
will be available at
https://github.com/tensorflow/models/tree/master/official/.
