Substitute Teacher Networks: Learning with Almost No Supervision
  Learning through experience is time-consuming, inefficient and often bad for
your cortisol levels. To address this problem, a number of recently proposed
teacher-student methods have demonstrated the benefits of private tuition, in
which a single model learns from an ensemble of more experienced tutors.
Unfortunately, the cost of such supervision restricts good representations to a
privileged minority. Unsupervised learning can be used to lower tuition fees,
but runs the risk of producing networks that require extracurriculum learning
to strengthen their CVs and create their own LinkedIn profiles. Inspired by the
logo on a promotional stress ball at a local recruitment fair, we make the
following three contributions. First, we propose a novel almost no supervision
training algorithm that is effective, yet highly scalable in the number of
student networks being supervised, ensuring that education remains affordable.
Second, we demonstrate our approach on a typical use case: learning to bake,
developing a method that tastily surpasses the current state of the art.
Finally, we provide a rigorous quantitive analysis of our method, proving that
we have access to a calculator. Our work calls into question the long-held
dogma that life is the best teacher.
