Pixel Distillation: A New Knowledge Distillation Scheme for
  Low-Resolution Image Recognition
  The great success of deep learning is mainly due to the large-scale network
architecture and the high-quality training data. However, it is still
challenging to deploy recent deep models on portable devices with limited
memory and imaging ability. Some existing works have engaged to compress the
model via knowledge distillation. Unfortunately, these methods cannot deal with
images with reduced image quality, such as the low-resolution (LR) images. To
this end, we make a pioneering effort to distill helpful knowledge from a heavy
network model learned from high-resolution (HR) images to a compact network
model that will handle LR images, thus advancing the current knowledge
distillation technique with the novel pixel distillation. To achieve this goal,
we propose a Teacher-Assistant-Student (TAS) framework, which disentangles
knowledge distillation into the model compression stage and the high resolution
representation transfer stage. By equipping a novel Feature Super Resolution
(FSR) module, our approach can learn lightweight network model that can achieve
similar accuracy as the heavy teacher model but with much fewer parameters,
faster inference speed, and lower-resolution inputs. Comprehensive experiments
on three widely-used benchmarks, \ie, CUB-200-2011, PASCAL VOC 2007, and
ImageNetSub, demonstrate the effectiveness of our approach.
