Temporally Coherent Embeddings for Self-Supervised Video Representation
  Learning
  This paper presents TCE: Temporally Coherent Embeddings for self-supervised
video representation learning. The proposed method exploits inherent structure
of unlabeled video data to explicitly enforce temporal coherency in the
embedding space, rather than indirectly learning it through ranking or
predictive proxy tasks. In the same way that high-level visual information in
the world changes smoothly, we believe that nearby frames in learned
representations will benefit from demonstrating similar properties. Using this
assumption, we train our TCE model to encode videos such that adjacent frames
exist close to each other and videos are separated from one another. Using TCE
we learn robust representations from large quantities of unlabeled video data.
We thoroughly analyse and evaluate our self-supervised learned TCE models on a
downstream task of video action recognition using multiple challenging
benchmarks (Kinetics400, UCF101, HMDB51). With a simple but effective 2D-CNN
backbone and only RGB stream inputs, TCE pre-trained representations outperform
all previous selfsupervised 2D-CNN and 3D-CNN pre-trained on UCF101. The code
and pre-trained models for this paper can be downloaded at:
https://github.com/csiro-robotics/TCE
