State of Compact Architecture Search For Deep Neural Networks
  The design of compact deep neural networks is a crucial task to enable
widespread adoption of deep neural networks in the real-world, particularly for
edge and mobile scenarios. Due to the time-consuming and challenging nature of
manually designing compact deep neural networks, there has been significant
recent research interest into algorithms that automatically search for compact
network architectures. A particularly interesting class of compact architecture
search algorithms are those that are guided by baseline network architectures.
Such algorithms have been shown to be significantly more computationally
efficient than unguided methods. In this study, we explore the current state of
compact architecture search for deep neural networks through both theoretical
and empirical analysis of four different state-of-the-art compact architecture
search algorithms: i) group lasso regularization, ii) variational dropout, iii)
MorphNet, and iv) Generative Synthesis. We examine these methods in detail
based on a number of different factors such as efficiency, effectiveness, and
scalability. Furthermore, empirical evaluations are conducted to compare the
efficacy of these compact architecture search algorithms across three
well-known benchmark datasets. While by no means an exhaustive exploration, we
hope that this study helps provide insights into the interesting state of this
relatively new area of research in terms of diversity and real, tangible gains
already achieved in architecture design improvements. Furthermore, the hope is
that this study would help in pushing the conversation forward towards a deeper
theoretical and empirical understanding where the research community currently
stands in the landscape of compact architecture search for deep neural
networks, and the practical challenges and considerations in leveraging such
approaches for operational usage.
