Spirit Distillation: Precise Real-time Semantic Segmentation of Road
  Scenes with Insufficient Data
  Semantic segmentation of road scenes is one of the key technologies for
realizing autonomous driving scene perception, and the effectiveness of deep
Convolutional Neural Networks(CNNs) for this task has been demonstrated.
State-of-art CNNs for semantic segmentation suffer from excessive computations
as well as large-scale training data requirement. Inspired by the ideas of
Fine-tuning-based Transfer Learning (FTT) and feature-based knowledge
distillation, we propose a new knowledge distillation method for cross-domain
knowledge transference and efficient data-insufficient network training, named
Spirit Distillation(SD), which allow the student network to mimic the teacher
network to extract general features, so that a compact and accurate student
network can be trained for real-time semantic segmentation of road scenes.
Then, in order to further alleviate the trouble of insufficient data and
improve the robustness of the student, an Enhanced Spirit Distillation (ESD)
method is proposed, which commits to exploit a more comprehensive general
features extraction capability by considering images from both the target and
the proximity domains as input. To our knowledge, this paper is a pioneering
work on the application of knowledge distillation to few-shot learning.
Persuasive experiments conducted on Cityscapes semantic segmentation with the
prior knowledge transferred from COCO2017 and KITTI demonstrate that our
methods can train a better student network (mIOU and high-precision accuracy
boost by 1.4% and 8.2% respectively, with 78.2% segmentation variance) with
only 41.8% FLOPs (see Fig. 1).
