Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
  Yes, they do. This paper provides the first empirical demonstration that deep
convolutional models really need to be both deep and convolutional, even when
trained with methods such as distillation that allow small or shallow models of
high accuracy to be trained. Although previous research showed that shallow
feed-forward nets sometimes can learn the complex functions previously learned
by deep nets while using the same number of parameters as the deep models they
mimic, in this paper we demonstrate that the same methods cannot be used to
train accurate models on CIFAR-10 unless the student models contain multiple
layers of convolution. Although the student models do not have to be as deep as
the teacher model they mimic, the students need multiple convolutional layers
to learn functions of comparable accuracy as the deep convolutional teacher.
