Small-Group Learning, with Application to Neural Architecture Search
  In human learning, an effective learning methodology is small-group learning:
a small group of students work together towards the same learning objective,
where they express their understanding of a topic to their peers, compare their
ideas, and help each other to trouble-shoot problems. In this paper, we aim to
investigate whether this human learning method can be borrowed to train better
machine learning models, by developing a novel ML framework -- small-group
learning (SGL). In our framework, a group of learners (ML models) with
different model architectures collaboratively help each other to learn by
leveraging their complementary advantages. Specifically, each learner uses its
intermediately trained model to generate a pseudo-labeled dataset and re-trains
its model using pseudo-labeled datasets generated by other learners. SGL is
formulated as a multi-level optimization framework consisting of three learning
stages: each learner trains a model independently and uses this model to
perform pseudo-labeling; each learner trains another model using datasets
pseudo-labeled by other learners; learners improve their architectures by
minimizing validation losses. An efficient algorithm is developed to solve the
multi-level optimization problem. We apply SGL for neural architecture search.
Results on CIFAR-100, CIFAR-10, and ImageNet demonstrate the effectiveness of
our method.
