torchdistill: A Modular, Configuration-Driven Framework for Knowledge
  Distillation
  While knowledge distillation (transfer) has been attracting attentions from
the research community, the recent development in the fields has heightened the
need for reproducible studies and highly generalized frameworks to lower
barriers to such high-quality, reproducible deep learning research. Several
researchers voluntarily published frameworks used in their knowledge
distillation studies to help other interested researchers reproduce their
original work. Such frameworks, however, are usually neither well generalized
nor maintained, thus researchers are still required to write a lot of code to
refactor/build on the frameworks for introducing new methods, models, datasets
and designing experiments. In this paper, we present our developed open-source
framework built on PyTorch and dedicated for knowledge distillation studies.
The framework is designed to enable users to design experiments by declarative
PyYAML configuration files, and helps researchers complete the recently
proposed ML Code Completeness Checklist. Using the developed framework, we
demonstrate its various efficient training strategies, and implement a variety
of knowledge distillation methods. We also reproduce some of their original
experimental results on the ImageNet and COCO datasets presented at major
machine learning conferences such as ICLR, NeurIPS, CVPR and ECCV, including
recent state-of-the-art methods. All the source code, configurations, log files
and trained model weights are publicly available at
https://github.com/yoshitomo-matsubara/torchdistill .
