On the Orthogonality of Knowledge Distillation with Other Techniques:
  From an Ensemble Perspective
  To put a state-of-the-art neural network to practical use, it is necessary to
design a model that has a good trade-off between the resource consumption and
performance on the test set. Many researchers and engineers are developing
methods that enable training or designing a model more efficiently. Developing
an efficient model includes several strategies such as network architecture
search, pruning, quantization, knowledge distillation, utilizing cheap
convolution, regularization, and also includes any craft that leads to a better
performance-resource trade-off. When combining these technologies together, it
would be ideal if one source of performance improvement does not conflict with
others. We call this property as the orthogonality in model efficiency. In this
paper, we focus on knowledge distillation and demonstrate that knowledge
distillation methods are orthogonal to other efficiency-enhancing methods both
analytically and empirically. Analytically, we claim that knowledge
distillation functions analogous to a ensemble method, bootstrap aggregating.
This analytical explanation is provided from the perspective of implicit data
augmentation property of knowledge distillation. Empirically, we verify
knowledge distillation as a powerful apparatus for practical deployment of
efficient neural network, and also introduce ways to integrate it with other
methods effectively.
