Benchmarks for Image Classification and Other High-dimensional Pattern
  Recognition Problems
  A good classification method should yield more accurate results than simple
heuristics. But there are classification problems, especially high-dimensional
ones like the ones based on image/video data, for which simple heuristics can
work quite accurately; the structure of the data in such problems is easy to
uncover without any sophisticated or computationally expensive method. On the
other hand, some problems have a structure that can only be found with
sophisticated pattern recognition methods. We are interested in quantifying the
difficulty of a given high-dimensional pattern recognition problem. We consider
the case where the patterns come from two pre-determined classes and where the
objects are represented by points in a high-dimensional vector space. However,
the framework we propose is extendable to an arbitrarily large number of
classes. We propose classification benchmarks based on simple random projection
heuristics. Our benchmarks are 2D curves parameterized by the classification
error and computational cost of these simple heuristics. Each curve divides the
plane into a "positive- gain" and a "negative-gain" region. The latter contains
methods that are ill-suited for the given classification problem. The former is
divided into two by the curve asymptote; methods that lie in the small region
under the curve but right of the asymptote merely provide a computational gain
but no structural advantage over the random heuristics. We prove that the curve
asymptotes are optimal (i.e. at Bayes error) in some cases, and thus no
sophisticated method can provide a structural advantage over the random
heuristics. Such classification problems, an example of which we present in our
numerical experiments, provide poor ground for testing new pattern
classification methods.
