An Efficient Algorithm for Bayesian Nearest Neighbours
  K-Nearest Neighbours (k-NN) is a popular classification and regression
algorithm, yet one of its main limitations is the difficulty in choosing the
number of neighbours. We present a Bayesian algorithm to compute the posterior
probability distribution for k given a target point within a data-set,
efficiently and without the use of Markov Chain Monte Carlo (MCMC) methods or
simulation - alongside an exact solution for distributions within the
exponential family. The central idea is that data points around our target are
generated by the same probability distribution, extending outwards over the
appropriate, though unknown, number of neighbours. Once the data is projected
onto a distance metric of choice, we can transform the choice of k into a
change-point detection problem, for which there is an efficient solution: we
recursively compute the probability of the last change-point as we move towards
our target, and thus de facto compute the posterior probability distribution
over k. Applying this approach to both a classification and a regression UCI
data-sets, we compare favourably and, most importantly, by removing the need
for simulation, we are able to compute the posterior probability of k exactly
and rapidly. As an example, the computational time for the Ripley data-set is a
few milliseconds compared to a few hours when using a MCMC approach.
