Regret Bounds for Non-decomposable Metrics with Missing Labels
  We consider the problem of recommending relevant labels (items) for a given
data point (user). In particular, we are interested in the practically
important setting where the evaluation is with respect to non-decomposable
(over labels) performance metrics like the $F_1$ measure, and the training data
has missing labels. To this end, we propose a generic framework that given a
performance metric $\Psi$, can devise a regularized objective function and a
threshold such that all the values in the predicted score vector above and only
above the threshold are selected to be positive. We show that the regret or
generalization error in the given metric $\Psi$ is bounded ultimately by
estimation error of certain underlying parameters. In particular, we derive
regret bounds under three popular settings: a) collaborative filtering, b)
multilabel classification, and c) PU (positive-unlabeled) learning. For each of
the above problems, we can obtain precise non-asymptotic regret bound which is
small even when a large fraction of labels is missing. Our empirical results on
synthetic and benchmark datasets demonstrate that by explicitly modeling for
missing labels and optimizing the desired performance metric, our algorithm
indeed achieves significantly better performance (like $F_1$ score) when
compared to methods that do not model missing label information carefully.
