Contrastive Neighborhood Alignment
  We present Contrastive Neighborhood Alignment (CNA), a manifold learning
approach to maintain the topology of learned features whereby data points that
are mapped to nearby representations by the source (teacher) model are also
mapped to neighbors by the target (student) model. The target model aims to
mimic the local structure of the source representation space using a
contrastive loss. CNA is an unsupervised learning algorithm that does not
require ground-truth labels for the individual samples. CNA is illustrated in
three scenarios: manifold learning, where the model maintains the local
topology of the original data in a dimension-reduced space; model distillation,
where a small student model is trained to mimic a larger teacher; and legacy
model update, where an older model is replaced by a more powerful one.
Experiments show that CNA is able to capture the manifold in a high-dimensional
space and improves performance compared to the competing methods in their
domains.
