Learning from Higher-Layer Feature Visualizations
  Driven by the goal to enable sleep apnea monitoring and machine
learning-based detection at home with small mobile devices, we investigate
whether interpretation-based indirect knowledge transfer can be used to create
classifiers with acceptable performance. Interpretation-based indirect
knowledge transfer means that a classifier (student) learns from a synthetic
dataset based on the knowledge representation from an already trained Deep
Network (teacher). We use activation maximization to generate visualizations
and create a synthetic dataset to train the student classifier. This approach
has the advantage that student classifiers can be trained without access to the
original training data. With experiments we investigate the feasibility of
interpretation-based indirect knowledge transfer and its limitations. The
student achieves an accuracy of 97.8% on MNIST (teacher accuracy: 99.3%) with a
similar smaller architecture to that of the teacher. The student classifier
achieves an accuracy of 86.1% and 89.5% for a subset of the Apnea-ECG dataset
(teacher: 89.5% and 91.1%, respectively).
