Introspective Learning by Distilling Knowledge from Online
  Self-explanation
  In recent years, many explanation methods have been proposed to explain
individual classifications of deep neural networks. However, how to leverage
the created explanations to improve the learning process has been less
explored. As the privileged information, the explanations of a model can be
used to guide the learning process of the model itself. In the community,
another intensively investigated privileged information used to guide the
training of a model is the knowledge from a powerful teacher model. The goal of
this work is to leverage the self-explanation to improve the learning process
by borrowing ideas from knowledge distillation. We start by investigating the
effective components of the knowledge transferred from the teacher network to
the student network. Our investigation reveals that both the responses in
non-ground-truth classes and class-similarity information in teacher's outputs
contribute to the success of the knowledge distillation. Motivated by the
conclusion, we propose an implementation of introspective learning by
distilling knowledge from online self-explanations. The models trained with the
introspective learning procedure outperform the ones trained with the standard
learning procedure, as well as the ones trained with different regularization
methods. When compared to the models learned from peer networks or teacher
networks, our models also show competitive performance and requires neither
peers nor teachers.
