AuxAdapt: Stable and Efficient Test-Time Adaptation for Temporally
  Consistent Video Semantic Segmentation
  In video segmentation, generating temporally consistent results across frames
is as important as achieving frame-wise accuracy. Existing methods rely either
on optical flow regularization or fine-tuning with test data to attain temporal
consistency. However, optical flow is not always avail-able and reliable.
Besides, it is expensive to compute. Fine-tuning the original model in test
time is cost sensitive.
  This paper presents an efficient, intuitive, and unsupervised online
adaptation method, AuxAdapt, for improving the temporal consistency of most
neural network models. It does not require optical flow and only takes one pass
of the video. Since inconsistency mainly arises from the model's uncertainty in
its output, we propose an adaptation scheme where the model learns from its own
segmentation decisions as it streams a video, which allows producing more
confident and temporally consistent labeling for similarly-looking pixels
across frames. For stability and efficiency, we leverage a small auxiliary
segmentation network (AuxNet) to assist with this adaptation. More
specifically, AuxNet readjusts the decision of the original segmentation
network (Main-Net) by adding its own estimations to that of MainNet. At every
frame, only AuxNet is updated via back-propagation while keeping MainNet fixed.
We extensively evaluate our test-time adaptation approach on standard video
benchmarks, including Cityscapes, CamVid, and KITTI. The results demonstrate
that our approach provides label-wise accurate, temporally consistent, and
computationally efficient adaptation (5+ folds overhead reduction comparing to
state-of-the-art test-time adaptation methods).
