Improving Neural ODEs via Knowledge Distillation
  Neural Ordinary Differential Equations (Neural ODEs) construct the continuous
dynamics of hidden units using ordinary differential equations specified by a
neural network, demonstrating promising results on many tasks. However, Neural
ODEs still do not perform well on image recognition tasks. The possible reason
is that the one-hot encoding vector commonly used in Neural ODEs can not
provide enough supervised information. We propose a new training based on
knowledge distillation to construct more powerful and robust Neural ODEs
fitting image recognition tasks. Specially, we model the training of Neural
ODEs into a teacher-student learning process, in which we propose ResNets as
the teacher model to provide richer supervised information. The experimental
results show that the new training manner can improve the classification
accuracy of Neural ODEs by 24% on CIFAR10 and 5% on SVHN. In addition, we also
quantitatively discuss the effect of both knowledge distillation and time
horizon in Neural ODEs on robustness against adversarial examples. The
experimental analysis concludes that introducing the knowledge distillation and
increasing the time horizon can improve the robustness of Neural ODEs against
adversarial examples.
