Stochastic Negative Mining for Learning with Large Output Spaces
  We consider the problem of retrieving the most relevant labels for a given
input when the size of the output space is very large. Retrieval methods are
modeled as set-valued classifiers which output a small set of classes for each
input, and a mistake is made if the label is not in the output set. Despite its
practical importance, a statistically principled, yet practical solution to
this problem is largely missing. To this end, we first define a family of
surrogate losses and show that they are calibrated and convex under certain
conditions on the loss parameters and data distribution, thereby establishing a
statistical and analytical basis for using these losses. Furthermore, we
identify a particularly intuitive class of loss functions in the aforementioned
family and show that they are amenable to practical implementation in the large
output space setting (i.e. computation is possible without evaluating scores of
all labels) by developing a technique called Stochastic Negative Mining. We
also provide generalization error bounds for the losses in the family. Finally,
we conduct experiments which demonstrate that Stochastic Negative Mining yields
benefits over commonly used negative sampling approaches.
