Can a powerful neural network be a teacher for a weaker neural network?
  The transfer learning technique is widely used to learning in one context and
applying it to another, i.e. the capacity to apply acquired knowledge and
skills to new situations. But is it possible to transfer the learning from a
deep neural network to a weaker neural network? Is it possible to improve the
performance of a weak neural network using the knowledge acquired by a more
powerful neural network? In this work, during the training process of a weak
network, we add a loss function that minimizes the distance between the
features previously learned from a strong neural network with the features that
the weak network must try to learn. To demonstrate the effectiveness and
robustness of our approach, we conducted a large number of experiments using
three known datasets and demonstrated that a weak neural network can increase
its performance if its learning process is driven by a more powerful neural
network.
