Architecture Compression
  In this paper we propose a novel approach to model compression termed
Architecture Compression. Instead of operating on the weight or filter space of
the network like classical model compression methods, our approach operates on
the architecture space. A 1-D CNN encoder-decoder is trained to learn a mapping
from discrete architecture space to a continuous embedding and back.
Additionally, this embedding is jointly trained to regress accuracy and
parameter count in order to incorporate information about the architecture's
effectiveness on the dataset. During the compression phase, we first encode the
network and then perform gradient descent in continuous space to optimize a
compression objective function that maximizes accuracy and minimizes parameter
count. The final continuous feature is then mapped to a discrete architecture
using the decoder. We demonstrate the merits of this approach on visual
recognition tasks such as CIFAR-10, CIFAR-100, Fashion-MNIST and SVHN and
achieve a greater than 20x compression on CIFAR-10.
