Generative Teaching Networks: Accelerating Neural Architecture Search by
  Learning to Generate Synthetic Training Data
  This paper investigates the intriguing question of whether we can create
learning algorithms that automatically generate training data, learning
environments, and curricula in order to help AI agents rapidly learn. We show
that such algorithms are possible via Generative Teaching Networks (GTNs), a
general approach that is, in theory, applicable to supervised, unsupervised,
and reinforcement learning, although our experiments only focus on the
supervised case. GTNs are deep neural networks that generate data and/or
training environments that a learner (e.g. a freshly initialized neural
network) trains on for a few SGD steps before being tested on a target task. We
then differentiate through the entire learning process via meta-gradients to
update the GTN parameters to improve performance on the target task. GTNs have
the beneficial property that they can theoretically generate any type of data
or training environment, making their potential impact large. This paper
introduces GTNs, discusses their potential, and showcases that they can
substantially accelerate learning. We also demonstrate a practical and exciting
application of GTNs: accelerating the evaluation of candidate architectures for
neural architecture search (NAS), which is rate-limited by such evaluations,
enabling massive speed-ups in NAS. GTN-NAS improves the NAS state of the art,
finding higher performing architectures when controlling for the search
proposal mechanism. GTN-NAS also is competitive with the overall state of the
art approaches, which achieve top performance while using orders of magnitude
less computation than typical NAS methods. Speculating forward, GTNs may
represent a first step toward the ambitious goal of algorithms that generate
their own training data and, in doing so, open a variety of interesting new
research questions and directions.
