An Efficient Method of Training Small Models for Regression Problems
  with Knowledge Distillation
  Compressing deep neural network (DNN) models becomes a very important and
necessary technique for real-world applications, such as deploying those models
on mobile devices. Knowledge distillation is one of the most popular methods
for model compression, and many studies have been made on developing this
technique. However, those studies mainly focused on classification problems,
and very few attempts have been made on regression problems, although there are
many application of DNNs on regression problems. In this paper, we propose a
new formalism of knowledge distillation for regression problems. First, we
propose a new loss function, teacher outlier rejection loss, which rejects
outliers in training samples using teacher model predictions. Second, we
consider a multi-task network with two outputs: one estimates training labels
which is in general contaminated by noisy labels; And the other estimates
teacher model's output which is expected to modify the noise labels following
the memorization effects. By considering the multi-task network, training of
the feature extraction of student models becomes more effective, and it allows
us to obtain a better student model than one trained from scratch. We performed
comprehensive evaluation with one simple toy model: sinusoidal function, and
two open datasets: MPIIGaze, and Multi-PIE. Our results show consistent
improvement in accuracy regardless of the annotation error level in the
datasets.
