Learning to Benchmark: Determining Best Achievable Misclassification
  Error from Training Data
  We address the problem of learning to benchmark the best achievable
classifier performance. In this problem the objective is to establish
statistically consistent estimates of the Bayes misclassification error rate
without having to learn a Bayes-optimal classifier. Our learning to benchmark
framework improves on previous work on learning bounds on Bayes
misclassification rate since it learns the {\it exact} Bayes error rate instead
of a bound on error rate. We propose a benchmark learner based on an ensemble
of $\epsilon$-ball estimators and Chebyshev approximation. Under a smoothness
assumption on the class densities we show that our estimator achieves an
optimal (parametric) mean squared error (MSE) rate of $O(N^{-1})$, where $N$ is
the number of samples. Experiments on both simulated and real datasets
establish that our proposed benchmark learning algorithm produces estimates of
the Bayes error that are more accurate than previous approaches for learning
bounds on Bayes error probability.
