kNet: A Deep kNN Network To Handle Label Noise
  Deep Neural Networks require large amounts of labeled data for their
training. Collecting this data at scale inevitably causes label noise.Hence,the
need to develop learning algorithms that are robust to label noise. In recent
years, k Nearest Neighbors (kNN) emerged as a viable solution to this problem.
Despite its success, kNN is not without its problems. Mainly, it requires a
huge memory footprint to store all the training samples and it needs an
advanced data structure to allow for fast retrieval of the relevant examples,
given a query sample. We propose a neural network, termed kNet, that learns to
perform kNN. Once trained, we no longer need to store the training data, and
processing a query sample is a simple matter of inference. To use kNet, we
first train a preliminary network on the data set, and then train kNet on the
penultimate layer of the preliminary network.We find that kNet gives a smooth
approximation of kNN,and cannot handle the sharp label changes between samples
that kNN can exhibit. This indicates that currently kNet is best suited to
approximate kNN with a fairly large k. Experiments on two data sets show that
this is the regime in which kNN works best,and can therefore be replaced by
kNet.In practice, kNet consistently improve the results of all preliminary
networks, in all label noise regimes, by up to 3%.
