Perceptual Consistency in Video Segmentation
  In this paper, we present a novel perceptual consistency perspective on video
semantic segmentation, which can capture both temporal consistency and
pixel-wise correctness. Given two nearby video frames, perceptual consistency
measures how much the segmentation decisions agree with the pixel
correspondences obtained via matching general perceptual features. More
specifically, for each pixel in one frame, we find the most perceptually
correlated pixel in the other frame. Our intuition is that such a pair of
pixels are highly likely to belong to the same class. Next, we assess how much
the segmentation agrees with such perceptual correspondences, based on which we
derive the perceptual consistency of the segmentation maps across these two
frames. Utilizing perceptual consistency, we can evaluate the temporal
consistency of video segmentation by measuring the perceptual consistency over
consecutive pairs of segmentation maps in a video. Furthermore, given a
sparsely labeled test video, perceptual consistency can be utilized to aid with
predicting the pixel-wise correctness of the segmentation on an unlabeled
frame. More specifically, by measuring the perceptual consistency between the
predicted segmentation and the available ground truth on a nearby frame and
combining it with the segmentation confidence, we can accurately assess the
classification correctness on each pixel. Our experiments show that the
proposed perceptual consistency can more accurately evaluate the temporal
consistency of video segmentation as compared to flow-based measures.
Furthermore, it can help more confidently predict segmentation accuracy on
unlabeled test frames, as compared to using classification confidence alone.
Finally, our proposed measure can be used as a regularizer during the training
of segmentation models, which leads to more temporally consistent video
segmentation while maintaining accuracy.
