Candidates vs. Noises Estimation for Large Multi-Class Classification
  Problem
  This paper proposes a method for multi-class classification problems, where
the number of classes K is large. The method, referred to as Candidates vs.
Noises Estimation (CANE), selects a small subset of candidate classes and
samples the remaining classes. We show that CANE is always consistent and
computationally efficient. Moreover, the resulting estimator has low
statistical variance approaching that of the maximum likelihood estimator, when
the observed label belongs to the selected candidates with high probability. In
practice, we use a tree structure with leaves as classes to promote fast beam
search for candidate selection. We further apply the CANE method to estimate
word probabilities in learning large neural language models. Extensive
experimental results show that CANE achieves better prediction accuracy over
the Noise-Contrastive Estimation (NCE), its variants and a number of the
state-of-the-art tree classifiers, while it gains significant speedup compared
to standard O(K) methods.
