Sub-Architecture Ensemble Pruning in Neural Architecture Search
  Neural architecture search (NAS) is gaining more and more attention in recent
years due to its flexibility and remarkable capability to reduce the burden of
neural network design. To achieve better performance, however, the searching
process usually costs massive computations that might not be affordable for
researchers and practitioners. While recent attempts have employed ensemble
learning methods to mitigate the enormous computational cost, however, they
neglect a key property of ensemble methods, namely diversity, which leads to
collecting more similar sub-architectures with potential redundancy in the
final design. To tackle this problem, we propose a pruning method for NAS
ensembles called "Sub-Architecture Ensemble Pruning in Neural Architecture
Search (SAEP)." It targets to leverage diversity and to achieve sub-ensemble
architectures at a smaller size with comparable performance to ensemble
architectures that are not pruned. Three possible solutions are proposed to
decide which sub-architectures to prune during the searching process.
Experimental results exhibit the effectiveness of the proposed method by
largely reducing the number of sub-architectures without degrading the
performance.
