AutoShrink: A Topology-aware NAS for Discovering Efficient Neural
  Architecture
  Resource is an important constraint when deploying Deep Neural Networks
(DNNs) on mobile and edge devices. Existing works commonly adopt the cell-based
search approach, which limits the flexibility of network patterns in learned
cell structures. Moreover, due to the topology-agnostic nature of existing
works, including both cell-based and node-based approaches, the search process
is time consuming and the performance of found architecture may be sub-optimal.
To address these problems, we propose AutoShrink, a topology-aware Neural
Architecture Search(NAS) for searching efficient building blocks of neural
architectures. Our method is node-based and thus can learn flexible network
patterns in cell structures within a topological search space. Directed Acyclic
Graphs (DAGs) are used to abstract DNN architectures and progressively optimize
the cell structure through edge shrinking. As the search space intrinsically
reduces as the edges are progressively shrunk, AutoShrink explores more
flexible search space with even less search time. We evaluate AutoShrink on
image classification and language tasks by crafting ShrinkCNN and ShrinkRNN
models. ShrinkCNN is able to achieve up to 48% parameter reduction and save 34%
Multiply-Accumulates (MACs) on ImageNet-1K with comparable accuracy of
state-of-the-art (SOTA) models. Specifically, both ShrinkCNN and ShrinkRNN are
crafted within 1.5 GPU hours, which is 7.2x and 6.7x faster than the crafting
time of SOTA CNN and RNN models, respectively.
