Optimal Sampling Density for Nonparametric Regression
  We propose a novel active learning strategy for regression, which is
model-agnostic, robust against model mismatch, and interpretable. Assuming that
a small number of initial samples are available, we derive the optimal training
density that minimizes the generalization error of local polynomial smoothing
(LPS) with its kernel bandwidth tuned locally: We adopt the mean integrated
squared error (MISE) as a generalization criterion, and use the asymptotic
behavior of the MISE as well as the locally optimal bandwidths (LOB) - the
bandwidth function that minimizes MISE in the asymptotic limit. The asymptotic
expression of our objective then reveals the dependence of the MISE on the
training density, enabling analytic minimization. As a result,we obtain the
optimal training density in a closed-form. The almost model-free nature of our
approach thus helps to encode the essential properties of the target problem,
providing a robust and model-agnostic active learning strategy. Furthermore,
the obtained training density factorizes the influence of local function
complexity, noise level and test density in a transparent and interpretable
way. We validate our theory in numerical simulations, and show that the
proposed active learning method outperforms the existing state-of-the-art
model-agnostic approaches.
