Temporally-Weighted Hierarchical Clustering for Unsupervised Action
  Segmentation
  Action segmentation refers to inferring boundaries of semantically consistent
visual concepts in videos and is an important requirement for many video
understanding tasks. For this and other video understanding tasks, supervised
approaches have achieved encouraging performance but require a high volume of
detailed frame-level annotations. We present a fully automatic and unsupervised
approach for segmenting actions in a video that does not require any training.
Our proposal is an effective temporally-weighted hierarchical clustering
algorithm that can group semantically consistent frames of the video. Our main
finding is that representing a video with a 1-nearest neighbor graph by taking
into account the time progression is sufficient to form semantically and
temporally consistent clusters of frames where each cluster may represent some
action in the video. Additionally, we establish strong unsupervised baselines
for action segmentation and show significant performance improvements over
published unsupervised methods on five challenging action segmentation
datasets. Our code is available at
https://github.com/ssarfraz/FINCH-Clustering/tree/master/TW-FINCH
