Warm-starting DARTS using meta-learning
Neural architecture search (NAS) has shown great promise in the field of
automated machine learning (AutoML). NAS has outperformed hand-designed
networks and made a significant step forward in the field of automating the
design of deep neural networks, thus further reducing the need for human
expertise. However, most research is done targeting a single specific task,
leaving research of NAS methods over multiple tasks mostly overlooked.
Generally, there exist two popular ways to find an architecture for some novel
task. Either searching from scratch, which is ineffective by design, or
transferring discovered architectures from other tasks, which provides no
performance guarantees and is probably not optimal. In this work, we present a
meta-learning framework to warm-start Differentiable architecture search
(DARTS). DARTS is a NAS method that can be initialized with a transferred
architecture and is able to quickly adapt to new tasks. A task similarity
measure is used to determine which transfer architecture is selected, as
transfer architectures found on similar tasks will likely perform better.
Additionally, we employ a simple meta-transfer architecture that was learned
over multiple tasks. Experiments show that warm-started DARTS is able to find
competitive performing architectures while reducing searching costs on average
by 60%.