Efficient Architecture Search by Network Transformation
  Techniques for automatically designing deep neural network architectures such
as reinforcement learning based approaches have recently shown promising
results. However, their success is based on vast computational resources (e.g.
hundreds of GPUs), making them difficult to be widely used. A noticeable
limitation is that they still design and train each network from scratch during
the exploration of the architecture space, which is highly inefficient. In this
paper, we propose a new framework toward efficient architecture search by
exploring the architecture space based on the current network and reusing its
weights. We employ a reinforcement learning agent as the meta-controller, whose
action is to grow the network depth or layer width with function-preserving
transformations. As such, the previously validated networks can be reused for
further exploration, thus saves a large amount of computational cost. We apply
our method to explore the architecture space of the plain convolutional neural
networks (no skip-connections, branching etc.) on image benchmark datasets
(CIFAR-10, SVHN) with restricted computational resources (5 GPUs). Our method
can design highly competitive networks that outperform existing networks using
the same design scheme. On CIFAR-10, our model without skip-connections
achieves 4.23\% test error rate, exceeding a vast majority of modern
architectures and approaching DenseNet. Furthermore, by applying our method to
explore the DenseNet architecture space, we are able to achieve more accurate
networks with fewer parameters.
