Compressing Models with Few Samples: Mimicking then Replacing
  Few-sample compression aims to compress a big redundant model into a small
compact one with only few samples. If we fine-tune models with these limited
few samples directly, models will be vulnerable to overfit and learn almost
nothing. Hence, previous methods optimize the compressed model layer-by-layer
and try to make every layer have the same outputs as the corresponding layer in
the teacher model, which is cumbersome. In this paper, we propose a new
framework named Mimicking then Replacing (MiR) for few-sample compression,
which firstly urges the pruned model to output the same features as the
teacher's in the penultimate layer, and then replaces teacher's layers before
penultimate with a well-tuned compact one. Unlike previous layer-wise
reconstruction methods, our MiR optimizes the entire network holistically,
which is not only simple and effective, but also unsupervised and general. MiR
outperforms previous methods with large margins. Codes will be available soon.
