Learning by Teaching, with Application to Neural Architecture Search
  In human learning, an effective skill in improving learning outcomes is
learning by teaching: a learner deepens his/her understanding of a topic by
teaching this topic to others. In this paper, we aim to borrow this
teaching-driven learning methodology from humans and leverage it to train more
performant machine learning models, by proposing a novel ML framework referred
to as learning by teaching (LBT). In the LBT framework, a teacher model
improves itself by teaching a student model to learn well. Specifically, the
teacher creates a pseudo-labeled dataset and uses it to train a student model.
Based on how the student performs on a validation dataset, the teacher
re-learns its model and re-teaches the student until the student achieves great
validation performance. Our framework is based on three-level optimization
which contains three stages: teacher learns; teacher teaches student; teacher
re-learns based on how well the student performs. A simple but efficient
algorithm is developed to solve the three-level optimization problem. We apply
LBT to search neural architectures on CIFAR-10, CIFAR-100, and ImageNet. The
efficacy of our method is demonstrated in various experiments.
