Hierarchically Decoupled Spatial-Temporal Contrast for Self-supervised
  Video Representation Learning
  We present a novel technique for self-supervised video representation
learning by: (a) decoupling the learning objective into two contrastive
subtasks respectively emphasizing spatial and temporal features, and (b)
performing it hierarchically to encourage multi-scale understanding. Motivated
by their effectiveness in supervised learning, we first introduce
spatial-temporal feature learning decoupling and hierarchical learning to the
context of unsupervised video learning. We show by experiments that
augmentations can be manipulated as regularization to guide the network to
learn desired semantics in contrastive learning, and we propose a way for the
model to separately capture spatial and temporal features at multiple scales.
We also introduce an approach to overcome the problem of divergent levels of
instance invariance at different hierarchies by modeling the invariance as loss
weights for objective re-weighting. Experiments on downstream action
recognition benchmarks on UCF101 and HMDB51 show that our proposed
Hierarchically Decoupled Spatial-Temporal Contrast (HDC) makes substantial
improvements over directly learning spatial-temporal features as a whole and
achieves competitive performance when compared with other state-of-the-art
unsupervised methods. Code will be made available.
