FLASH: Fast Neural Architecture Search with Hardware Optimization
  Neural architecture search (NAS) is a promising technique to design efficient
and high-performance deep neural networks (DNNs). As the performance
requirements of ML applications grow continuously, the hardware accelerators
start playing a central role in DNN design. This trend makes NAS even more
complicated and time-consuming for most real applications. This paper proposes
FLASH, a very fast NAS methodology that co-optimizes the DNN accuracy and
performance on a real hardware platform. As the main theoretical contribution,
we first propose the NN-Degree, an analytical metric to quantify the
topological characteristics of DNNs with skip connections (e.g., DenseNets,
ResNets, Wide-ResNets, and MobileNets). The newly proposed NN-Degree allows us
to do training-free NAS within one second and build an accuracy predictor by
training as few as 25 samples out of a vast search space with more than 63
billion configurations. Second, by performing inference on the target hardware,
we fine-tune and validate our analytical models to estimate the latency, area,
and energy consumption of various DNN architectures while executing standard ML
datasets. Third, we construct a hierarchical algorithm based on simplicial
homology global optimization (SHGO) to optimize the model-architecture
co-design process, while considering the area, latency, and energy consumption
of the target hardware. We demonstrate that, compared to the state-of-the-art
NAS approaches, our proposed hierarchical SHGO-based algorithm enables more
than four orders of magnitude speedup (specifically, the execution time of the
proposed algorithm is about 0.1 seconds). Finally, our experimental evaluations
show that FLASH is easily transferable to different hardware architectures,
thus enabling us to do NAS on a Raspberry Pi-3B processor in less than 3
seconds.
