Deep geometric knowledge distillation with graphs
  In most cases deep learning architectures are trained disregarding the amount
of operations and energy consumption. However, some applications, like embedded
systems, can be resource-constrained during inference. A popular approach to
reduce the size of a deep learning architecture consists in distilling
knowledge from a bigger network (teacher) to a smaller one (student). Directly
training the student to mimic the teacher representation can be effective, but
it requires that both share the same latent space dimensions. In this work, we
focus instead on relative knowledge distillation (RKD), which considers the
geometry of the respective latent spaces, allowing for dimension-agnostic
transfer of knowledge. Specifically we introduce a graph-based RKD method, in
which graphs are used to capture the geometry of latent spaces. Using classical
computer vision benchmarks, we demonstrate the ability of the proposed method
to efficiently distillate knowledge from the teacher to the student, leading to
better accuracy for the same budget as compared to existing RKD alternatives.
