Improving the Interpretability of Deep Neural Networks with Knowledge
  Distillation
  Deep Neural Networks have achieved huge success at a wide spectrum of
applications from language modeling, computer vision to speech recognition.
However, nowadays, good performance alone is not sufficient to satisfy the
needs of practical deployment where interpretability is demanded for cases
involving ethics and mission critical applications. The complex models of Deep
Neural Networks make it hard to understand and reason the predictions, which
hinders its further progress. To tackle this problem, we apply the Knowledge
Distillation technique to distill Deep Neural Networks into decision trees in
order to attain good performance and interpretability simultaneously. We
formulate the problem at hand as a multi-output regression problem and the
experiments demonstrate that the student model achieves significantly better
accuracy performance (about 1\% to 5\%) than vanilla decision trees at the same
level of tree depth. The experiments are implemented on the TensorFlow platform
to make it scalable to big datasets. To the best of our knowledge, we are the
first to distill Deep Neural Networks into vanilla decision trees on
multi-class datasets.
