Evaluation Metrics for Item Recommendation under Sampling
  The task of item recommendation requires ranking a large catalogue of items
given a context. Item recommendation algorithms are evaluated using ranking
metrics that depend on the positions of relevant items. To speed up the
computation of metrics, recent work often uses sampled metrics where only a
smaller set of random items and the relevant items are ranked. This paper
investigates sampled metrics in more detail and shows that sampled metrics are
inconsistent with their exact version. Sampled metrics do not persist relative
statements, e.g., 'algorithm A is better than B', not even in expectation.
Moreover the smaller the sampling size, the less difference between metrics,
and for very small sampling size, all metrics collapse to the AUC metric.
