Moonshine: Distilling with Cheap Convolutions
  Many engineers wish to deploy modern neural networks in memory-limited
settings; but the development of flexible methods for reducing memory use is in
its infancy, and there is little knowledge of the resulting cost-benefit. We
propose structural model distillation for memory reduction using a strategy
that produces a student architecture that is a simple transformation of the
teacher architecture: no redesign is needed, and the same hyperparameters can
be used. Using attention transfer, we provide Pareto curves/tables for
distillation of residual networks with four benchmark datasets, indicating the
memory versus accuracy payoff. We show that substantial memory savings are
possible with very little loss of accuracy, and confirm that distillation
provides student network performance that is better than training that student
architecture directly on data.
