Sobolev Training for Neural Networks
  At the heart of deep learning we aim to use neural networks as function
approximators - training them to produce outputs from inputs in emulation of a
ground truth function or data creation process. In many cases we only have
access to input-output pairs from the ground truth, however it is becoming more
common to have access to derivatives of the target output with respect to the
input - for example when the ground truth function is itself a neural network
such as in network compression or distillation. Generally these target
derivatives are not computed, or are ignored. This paper introduces Sobolev
Training for neural networks, which is a method for incorporating these target
derivatives in addition the to target values while training. By optimising
neural networks to not only approximate the function's outputs but also the
function's derivatives we encode additional information about the target
function within the parameters of the neural network. Thereby we can improve
the quality of our predictors, as well as the data-efficiency and
generalization capabilities of our learned function approximation. We provide
theoretical justifications for such an approach as well as examples of
empirical evidence on three distinct domains: regression on classical
optimisation datasets, distilling policies of an agent playing Atari, and on
large-scale applications of synthetic gradients. In all three domains the use
of Sobolev Training, employing target derivatives in addition to target values,
results in models with higher accuracy and stronger generalisation.
