{'fraction_non_alphanumeric': 0.02809159591139803, 'fraction_numerical': 0.00914555703693308, 'mean_word_length': 4.948796438013079, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
New Insights into Metric Optimization for Ranking-based Recommendation

Direct optimization of IR metrics has often been adopted as an approach to devise and develop ranking-based recommender systems. Most methods following this approach aim at optimizing the same metric being used for evaluation, under the assumption that this will lead to the best performance. A number of studies of this practice bring this assumption, however, into question. In this paper, we dig deeper into this issue in order to learn more about the effects of the choice of the metric to optimize on the performance of a ranking-based recommender system. We present an extensive experimental study conducted on different datasets in both pairwise and listwise learning-to-rank scenarios, to compare the relative merit of four popular IR metrics, namely RR, AP, nDCG and RBP, when used for optimization and assessment of recommender systems in various combinations. For the first three, we follow the practice of loss function formulation available in literature. For the fourth one, we propose novel loss functions inspired by RBP for both the pairwise and listwise scenario. Our results confirm that the best performance is indeed not necessarily achieved when optimizing the same metric being used for evaluation. In fact, we find that RBP-inspired losses perform at least as well as other metrics in a consistent way, and offer clear benefits in several cases. Interesting to see is that RBP-inspired losses, while improving the recommendation performance for all uses, may lead to an individual performance gain that is correlated with the activity level of a user in interacting with items. The more active the users, the more they benefit. Overall, our results challenge the assumption behind the current research practice of optimizing and evaluating the same metric, and point to RBP-based optimization instead as a promising alternative when learning to rank in the recommendation context.


INTRODUCTION
Offline evaluation of ranking-based recommender systems generally relies on effectiveness metrics from Information Retrieval (IR). These metrics quantify the quality of a ranked list of items in terms of their relevance [44] and according to the particular evaluation criteria they capture. Therefore, it has been seen as an intuitive and logical choice to learn a ranking model for a recommender system by directly optimizing the metric used for evaluation [26]. Different ranking approaches have been designed along this line, aiming at achieving a better recommendation performance [25,[39][40][41].
Previous research shows, however, that optimizing the metric used for evaluation is not necessarily the best approach. Different IR metrics reflect different aspects of retrieval performance [1,21,28,36], and do so to different extents. Although no metric covers all evaluation criteria, there is evidence that some metrics are more informative than others [3,4,49].This may enable them to, when used for optimization, achieve the best performance in view of a given evaluation metric other than itself, or even in view of multiple target evaluation criteria. Empirical results show that the advantage in informativeness can indeed be transferred into higher effectiveness when these metrics are used for optimization. Results on web search [12] and text retrieval [49] show that, when targeting at less informative metrics such as @10 and , optimizing more informative metrics, like or , can perform even better than optimizing the less informative metrics themselves.
The discussion above points to the possibility to achieve better ranking performance according to the evaluation criterion captured by metric if we do not optimize for , but for a more informative metric instead. This, however, is more likely to be successful when the metrics an are highly correlated. From the perspective of ranking effectiveness, correlation would typically arise between metrics that share characteristics, such as top-weightedness and localization. According to Moffat [28], correlation in view of such properties could tell something about the ability of the metrics to reveal the same aspects of the system behavior, as opposed to non-correlated metrics that reveal different aspects of the system behavior. Previous research on text retrieval and web search has shown high level of correlation among many metrics, such as between and , and , or @ and [5,16,47]. When targeting at maximizing one specific evaluation metric, one should therefore search among informative metrics correlated to the evaluation target to find the most effective optimization objective.
Motivated by the above, in this paper we revisit the choice of the metric to optimize in a ranking-based recommendation scenario. Our goal is to provide more insights into this optimization scenario, broadening the possibilities to benefit from such a metric choice, compared to what is commonly reported in the literature.
Although several popular metrics like , and have been applied to the task, (Rank-Biased Precision) [31], another important effectiveness metric widely used in traditional IR tasks, has hardly been applied for training and/or testing recommendation models. Nonetheless, is an informative [2,28,29] and flexible metric that incorporates a simple user model through a persistence parameter. Since varying the value of the persistence parameter makes correlated with different groups of metrics [30,36], has the potential to optimize for different metrics that reflect different evaluation criteria [16,31].
Following the spirit of existing metric optimization processes, we propose novel objective functions inspired by for both the pairwise and listwise learning-to-rank (LTR) paradigms. In this way, we enable to join the other metrics and serve as both the learning objective and evaluation target for our investigation. Specifically for the listwise case, we will show that minimizing the proposed -based objective function provides an elegant instrument to directly optimize for the ranking positions of the relevant items. Furthermore, the proposed -based listwise loss function is independent of the persistence parameter , which makes it possible to conduct -based optimization using a single unified framework, regardless of the target persistence to evaluate.
Empirical results obtained on four real-world datasets point to the following main insights: (1) The assumption behind the practice to optimize and evaluate ranking-based recommender systems using the same metric does not necessarily lead to the best performance. (2) -inspired losses perform at least as well as other metrics in a consistent way, and offer clear benefits in several cases. This makes a promising alternative when learning to rank in the recommendation context.
-based listwise optimization leads -relative to other metrics-to a significantly better ranking performance for active users with more of the relevant interactions, compared to less active users. However, this performance bias does not come at the cost of reducing utility for inactive users. On the contrary, the ranking performance improves for all users, only to different degrees.
The remainder of this paper is organized as follows. In Section 2, we position our contribution in the context of the related previous work. Section 3 describes the technical details of the models we use for a direct optimization of IR metrics. Section 4 introduces the design and protocols of our experiments, the results of which we present and discuss in Section 5. Section 6 concludes the paper with pointers to future work.

RELATED WORK
Direct optimization of IR metrics is a logical way of building ranking-based recommenders. Despite the fact that almost any IR metric can be transformed into an objective function for optimization, the choice of metric to optimize for maximizing the ranking effectiveness remains non-trivial. Intuitively, choosing to optimize more informative metrics helps with achieving higher ranking effectiveness. Several studies [3,4,49], inspired by Shannon [38] and Jaynes [20], have assessed the informativeness of IR metrics by how well they constrain a maximum entropy distribution over the relevance of ranked items, in the sense that such distribution accurately estimates the precision-recall curve or other metric scores for the same items. The results indicate that more complex metrics, such as and , are more informative than simpler metrics [47].
One way of achieving optimal ranking with respect to a given metric is to deploy a pairwise LTR approach. The pairwise LTR paradigm considers relevant-irrelevant (positive-negative) item pairs, and aims at maximizing the change in the considered IR metric caused by a ranking position swap. This idea lays the foundation for a batch of models, including LambdaRank [7], LambdaMart [6,19] and LambdaFM [50]. In particular, LambdaRank is widely used as the underlying model in studies comparing the optimization of different metrics. LambdaRank-based results in [12,49] show that optimizing for informative metrics can lead to good performance, not only when evaluating with the same metric, but also with others. This insight invites to revisit pairwise learning recommender systems by experimenting with other metrics to optimize, even if they are not the evaluation target. Similarly, LambdaFM [50] was deployed to assess effectiveness with respect to three metrics, namely , and , by optimizing for . Optimal performance was achieved with respect to and . Another way of achieving optimal ranking with respect to a given metric is to deploy a listwise LTR approach. This approach looks at the entire ranked list for optimization, and therefore better resembles the concept of direct metric optimization than the (indirect) pairwise LTR approach. Although straightforward and close in nature to LTR, listwise methods have to deal with loss functions containing integer ranking positions, which causes nonsmoothness and therefore non-differentiability. A common way to deal with this problem is to approximate the indicator function by a differentiable alternative. CofiRank [48] was one of the first works addressing this issue by choosing to minimize the (1 − ) loss with a structured estimation. Another popular method is to use a smooth function, such as a sigmoid or ReLU [32], to approximate the non-smooth indicator function. This method has been widely applied for optimizing [25], [39] and [40]. Rather than optimizing the whole list and taking items at the bottom into account, Liang et al. [25] proposed Top-N-Rank, which focuses on the top ranked items and uses a listwise loss with a cutoff to directly optimize for @ . Despite this rich track record of attempts to learn a ranking by metric optimization, still insufficient is known about what metric to optimize for in order to obtain the best performance according to some evaluation metric. Moreover, we believe that the scope of the metrics to consider could further be expanded to broaden the possibilities for improving ranking effectiveness beyond what has been tried so far. In this paper, to conduct our experimental assessment regarding ranking effectiveness for recommendation, we follow both the pairwise and listwise LTR approaches, and consider different IR metrics to optimize and assess ranking performance. Specifically, we add the metric as a promising candidate to the set of typically deployed , and .

METHODS
In this section we describe the design choices and methodology behind our experimental approach to acquire new insights into the issues related to generating recommendations through optimizing IR metrics. We start by introducing our underlying recommendation model with the notation and terminology used throughout the paper. Then, we describe the four IR metrics we choose to optimize. Finally, we define the corresponding objective functions we deploy for optimization in both the pairwise and listwise cases. In this way, we put special emphasis on the definitions of objective functions for that we introduce in this paper.

Recommendation Model
Recommender systems are meant to recommend "items" (in a general meaning of the term) to users according to their preferences. For a system with users and items, ground-truth user-item interaction data can be represented by a matrix with dimensions × . We consider a binary relevance scenario in this paper, which implies that elements in are either = 1, indicating a positive interaction (preference) between a user and an item , or = 0, indicating either a negative interaction (e.g. a 'dislike'), or no interaction between and . We refer to items with the positive (negative) interaction as the positive (negative) items. We assume that an arbitrary user generated positive interactions across all items.
Following the practice from other ranking-based recommendation approaches that target direct metric optimization [23,25,40], in this paper we choose Matrix Factorization (MF) [35] as the recommendation model. Although collaborative filtering can be achieved via more advanced methods such as Neural Collaborative Filtering [18], Collaborative Variational Autoencoders [24] and Graph Neural Networks [11,17], we still choose the base Matrix Factorization model because our aim in this paper is to study the relative merits of metrics. A more comprehensive experiment to assess generalizability with other models is left for future work.
The users and items are thus represented by latent factor matrices × and × , respectively, where is the number of latent factors. Using the latent vectors of users and items, a recommendation model can predict the relevance of items for each user, and store the scores in the matrix × , with the element representing the predicted relevance of item to user . The ranking position corresponding to the relevance score , is an integer ranging from 1 to , calculated from a pairwise comparison between the predicted relevance score for item and all other items: where I(·) denotes the indicator function.

Metrics
As introduced before, we consider four metrics to optimize when training the ranking mechanism of a recommender system: , , and . These metrics, assessing the recommendation performance for user , can be formulated as follows: According to the formulation originally proposed by Moffat and Zobel [31], is a constant parameter ranging from 0 to 1, indicating the degree of persistence of a user. A high persistence models a user that is willing to explore items deep down the ranked list. The theoretical upper limit of is 1 when is infinite, which means 1 is never reached in practice [36]. To align the range of with other metrics used in the paper and in this way make it more comparable, we choose to optimize instead, which normalizes the bare by the maximum obtainable with positive items: where ( , ) = 1/(1 − ), serving as a normalization factor.

Pairwise Metric Optimization
Following the same rationale as in Section 3.1, we choose Lamb-daRank [7], the base Lambda gradient ranking model, as the pairwise LTR approach. We do not consider the more complex Lamb-daMART [6] and LambdaFM [50] to avoid the effect of other factors such as model ensemble and dynamic negative sampling strategy. Derived from RankNet [8], LambdaRank aims at obtaining smooth gradients for optimization by calculating the performance gain from swapping the position of documents and in a ranked list. For the -optimization of , and , we follow the existing approaches proposed by Donmez et al. [12]. To the best of our knowledge, LambdaRank using has not been formally proposed yet, so we define the -optimization of following the same principles.
The cost for -optimizing an item pair ( , ) for user is where equals +1 or −1 depending on whether the ground truth label is larger than that of , and the term ≡ − represents the difference of the predicted relevance scores. The derivative of the cost with respect to can be formulated as In order to reward the positive gains and punish the negative, the -gradient for can thus be written as where and are the ranking positions of the item pair, calculated as in Eq. (1), and Δ ( , ; ) is the difference between the corresponding nRBP values. This leads to the -basedgradient formulated as

Listwise Metric Optimization
Pairwise methods, such as LambdaRank, can easily avoid the issue of non-smoothness of the optimized ranking metrics. However, and despite their success, eluding this issue in the listwise approach remains an open problem [12]. Successfully addressing this challenge is important because, in that way, the optimization process becomes more intuitive, straightforward and natural [27]. As mentioned in Section 2, methods have already been proposed for listwise optimization of , and . The underlying principle is to approximate the non-differentiable indicator function with a smooth alternative. This has been done by deploying either a sigmoid function [39,40] or the Rectified Linear Unit (ReLU) [32] as proposed in [25]. To make a consistent comparison, in this paper we use a sigmoid for all metrics.
Apart from the difference in the choice of the smoothing function, compared to TFMAP [39] and CLiMF [40], Top-N-Rank [25] is also distinctive for the way the approximated ranking position˜is modeled. TFMAP and CLiMF model˜using only the predicted score . In contrast, Top-N-Rank infers the predicted ranking position by looking at the whole recommendation list. It follows the idea from Eq. (1) to get the ranking position by pairwise comparison, which is closer in nature to sorting.
Taking all the above into account, and following the same rationale as in Sections 3.1 and 3.3, we do not contemplate more complex techniques such as Boosting [15,45] or multi-agent learning [52], so that metrics are compared on a base recommendation model derived from Top-N-Rank. Specifically, we replace ReLU by a sigmoid function and approximate the ranking position as where ( ) = 1/(1 + − ). Accordingly, the approximation of the indicator functions in Eqs. (3) and (4) can be formulated as Top-N-Rank, however, optimizes metrics with a cutoff, so it does not use information from all items. As indicated by Donmez et al. [12], optimizing on the whole item list can lead to significantly better @10 performance than directly optimizing @10. Consequently, we choose to eliminate the cutoff. Since the target is to maximize the approximated IR metrics, we can consider their additive inverse as the loss functions for optimization. Based on the above, the , and loss functions for user can be formulated as follows: To the best of our knowledge, no method has been proposed yet for -based listwise optimization. Inspired by the loss definitions for the other metrics, and again choosing to work with the normalized formulation of , we introduce the method for defining the corresponding loss function as follows.
By virtue of the monotonicity of the logarithm function, the recommendation model that optimizes ( ; ) also optimizes Note that the second term is a constant for each user, so we focus on the first term. Based on Jensen's inequality, we can derive a lower bound for the first term as follows: Note that the last equality holds because only = 1 contributes to the summation, and the two remaining logarithmic terms are constant across users.
Because ∈ (0, 1), ln( ) is negative, so maximizing the formulation in Eq. (18) becomes equivalent to minimizing =1 (˜− 1). In this way, our -based optimization of the ranking treats all relevant items equally and aims at bringing them close to the top. Although convenient and intuitive, this function does not have common bounds across users. Therefore, if used alone as the optimization objective, it would make the training process sensitive to specific users. To (partially) resolve this issue, we come back to the second term in Eq. (17), find its own "lower bound" using Jensen's inequality and subtract it from Eq. (18) to make the lower bound equal 0 for all users. After dropping the logarithms, the regulated loss for user can now be denoted as In this way, the optimization of becomes equivalent to an elegant direct optimization for the rank position of the relevant items. In view of the fact that the ideal situation leads to ranking all relevant items at the top, the listwise loss inspired by shows potential for achieving high ranking effectiveness across different evaluation criteria, making it an informative metric. Furthermore, is independent of , which makes it possible to conduct based optimization for different values in one unified framework. We note here once again that the regularization in Eq. (19) only has effect on the lower bound of the loss value range. The loss value remains user-sensitive and can still be arbitrarily large depending on the number of interactions . We analyze the consequences of this in Section 5.3.

Model Learning
In recent years, Adaptive Moment Estimation (Adam) [22] has become one of the most popular optimizers. Compared to traditional optimizers like Stochastic Gradient Descent (SGD), its insensitivity to hyper-parameters and faster convergence makes it widely deployed in machine learning models. Despite these advantages, Adam tends to suffer from convergence and generalization power [29,51]. Because in this paper we investigate the generalization power of optimizing for different metrics, we still choose to optimize all our models with SGD.

EXPERIMENTAL DESIGN
Our goal is to investigate the capabilities of metrics used for optimization when the recommendation performance is assessed by the same or other metrics. In the following we explain the selection of datasets and experimental protocol for our experiments.

Datasets
We selected four widely-used and real-world datasets to experiment with a diverse set of data. Two of them, CiteULike-a [46] and Epinions [43], contain unary data, while the other two, Sports & Outdoors and Home & Kitchen, are datasets with graded ratings from Amazon [33]. Amazon datasets contain integer relevance scores ranging from 1 to 5, so we need to binarize them before they can be used by our LTR methods. We choose to consider as positive only ratings of 4 and 5, which surely reflect a positive preference, and every other rating as a negative preference. In addition, and as is common in experimentation with recommenders, the absence of a rating is also taken as a negative interaction in all datasets [42]. As shown by Cañamares et al. [10], although users with few ratings might exist in commercial services, they are usually filtered out in offline experiments because the lack of data leads to unreliable performance measurements. To address this issue, we only keep users with at least 25 relevant interactions in all datasets. The post-processed dataset statistics are shown in Table 1.

Experimental Protocol
We use LensKit [14] to randomly split the data into training and test sets , stratifying by user: we sample 80% of their interactions for training, and leave the rest for testing. As a result, each user has at least 20 relevant interactions in the training set, and at least 5 in the test set. The evaluation metrics used are , , and with equal to 0.8, 0.9 and 0.95, so that we can assess recommendation performance under different degrees of user persistence. For pairwise optimization using LambdaRank, these six metrics are in line with six separate loss functions. In the listwise context, however, we have in total 4 loss functions because the loss for is independent of . All models are trained with a considerable number of epochs (3,000), so that every metric has a more than reasonable chance to achieve its best performance. For each model, we select the epoch yielding the best performance on the corresponding evaluation metric. For the optimization of listwise , the optimal epoch is chosen for each of the 3 values of separately, so that we actually have 3 different models.
To reduce random error due to data splitting, we adopt a Monte Carlo cross-validation approach [13] and create three independent splits per dataset. Shi et al. [39] indicate that, for IR metrics that only rely on the ranking positions of relevant items, there is no need to consider all irrelevant items when training. As a result, a negative sampling process can significantly speed up training without hurting the overall performance. Negative sampling is, however, not only beneficial for efficiency. According to Cañamares and Castells [9], removing some (or even a significant number of) negative items from both the training and test sets can make the evaluation more informative and less biased with, balancing popularity and the average relevance of items across users. Such a strategy can therefore also make the evaluation more effective. We choose to inform the negative sampling process by the number of relevant items for each user, so we sample as many negative items as positives, twice as many, or five times as many; we denote this as the Negative Sampling Ratio (NSR). These negative items, along with the 80% of positives, form the complete training set for a user. In order to align the distributions of training and test sets, we complete the test set using the same approach: the remaining 20% of positives, plus 100%, 200% or 500% as many negatives.
In order to maximize the performance of the assessed models, we fine-tuned the learning rate of SGD and performed a search in the range {0.001, 0.01, 0.1} for LambdaRank and {0.001, 0.01, 0.1, 1, 3, 10} for listwise models. We also conducted a preliminary exploration on the number of latent factors for Matrix Factorization within the range {8, 16, 32, 64, 128}. The results showed that, although there is a positive correlation between the ranking effectiveness and the latent space dimensionality, such a correlation has no impact on the relative performance of different losses. Therefore, we do not analyze the effect of dimensionality in this paper and simply fix the number of latent factors at 32 throughout the experiments.
We implement all models in PyTorch [34]. To accelerate the training process, we use CUDA and CuDNN on an NVIDIA GeForce GTX 1080Ti GPU.

RESULTS
In this section we compare the effectiveness of different metrics when used for optimization in ranking-based recommender. The goal of the analysis is threefold. First, in Section 5.1 we focus on  Figure 1: Overall performance of pairwise and listwise methods. metric_optim denotes the metric-based losses for optimization, and a panel row denotes an evaluation metric. All results are averaged over the 3 data splits. Note that y-axes vary per row.
the overall performance of pairwise and listwise models and investigate whether the practice of optimizing for the metric used in evaluation is justified in ranking-based recommendation. Then, in Section 5.2 we conduct a deeper analysis on the impact of a metric chosen for optimization on the ranking effectiveness assessed by different evaluation metrics. In doing so, we especially focus on the performance of the -inspired objective functions introduced in this paper. Finally, in Section 5.3 we investigate the effect of different metric optimization strategies on the recommendation utility for active and inactive users, with special emphasis on the impact of the missing upper bound of the -based listwise loss function proposed in Eq. (19).
Due to space constraints, we do not report all the results obtained in our experiments. The reported results are, however, fully representative of the complete set of results that led to the final observations, conclusions and recommendations for future work. 1

Should we Optimize the Metric Used to
Evaluate? Fig. 1 shows the performance of all pairwise and listwise learning objectives on all 6 evaluation metrics. While a negative correlation can be observed between recommendation effectiveness and the NSR, this does not necessarily mean that the models trained with more irrelevant items are worse. Because the training and test sets follow the same distribution, more negative items in the test set just make the relevance prediction task harder. Several observations can be made from this figure. First, in both pairwise and listwise models, optimizing consistently yields significantly worse performance, even when the evaluation target is also . This observation supports previous findings that 1 All data, code and full results are available at https://github.com/roger-zhe-li/sigir21-newinsights. optimizing other metrics can achieve higher test scores than optimizing itself [12]. The explanation for this is two-fold. First, does not exploit all the information in the training data because it only focuses on the first relevant item, resulting in suboptimal models compared to optimizing other metrics. Second, as indicated by Webber et al. [47], is not well correlated with other informative metrics, such as and , which may optimize for but not the other way around. Because the performance gap of optimizing is stable and significant, we do not include it in further analysis and instead focus on the other 5 learning objectives (metric_optim), but still with all 6 evaluation metrics.
Second, listwise and pairwise methods behave differently when optimizing . In the pairwise context, the recommendation performance obtained by optimizing different losses is varied. Specifically, we find that optimizing with = 0.95 outperforms optimizing with = 0.8 or = 0.9. This finding is consistent with prior research showing that .95 is better correlated with informative metrics than with other different 's [30,36]. Such an inneradvantage can also be explained by the nature of the metric. Because models user persistence, a high value takes relevant items lying deeper in the list into account during training. This allows the model to account for more information, which benefits its performance. Although higher also brings a slower weight decay, which does not favor prediction for the top of a recommendation list, this effect is insignificant for systems with binary relevance, where we do not need to put highly relevant items ahead of moderately relevant ones. In the listwise context, however, the results obtained when optimizing for with different values are in general homogeneous. This shows that our -independent loss provides a generic method to optimize for multiple metrics. In this way, the concern of choosing a specific for optimization is addressed in a simple and effective way.  Figure 2: The frequency for each metric-based loss (metric_optim) of achieving the best performance on specific evaluation metrics (metric_eval), in all 3 data splits and all 3 NSR's.
Last but not least, in both listwise and pairwise paradigms optimizing , and -inspired losses achieves similar stable performance across different datasets, NSRs and evaluation targets. Such an observation, combined with the finding that optimizing leads to the worst ranking effectiveness when evaluating with , suggests that the practice to optimize and evaluate recommender systems with the same metric is not necessarily the best. A more detailed analysis on the relative advantages of individual metrics is given in the next section.

Is
More Effective as Optimization Metric than Others? Fig. 2 shows how often each metric achieved the best test performance, across evaluation metrics, when used for optimization. Since frequencies are counted from 3 splits and 3 NSR's for each evaluation metric, the frequency is expected to have a row-wise sum of 9. Of course, it is possible for different losses to tie and obtain the best result for a certain case, so the row-wise sum is actually larger than 9 for several evaluation metrics, specially in the listwise case. Overall, we find that in both pairwise and listwise scenarios most of the best performance cases are achieved when optimizing inspired metrics. More importantly, in LambdaRank we even find that optimizing .95 shows a significant and clear advantage over all the other metrics.
Even though -based losses seem to achieve the best performance, Fig. 1 suggests that differences could be too small or indistinguishable from random error, so next we proceed to a statistical analysis. First, we standardize performance scores within dataset-NSR combination to avoid homoscedasticity, because Fig. 1 evidences very different scales per dataset and NSR. This way, scores are comparable across metrics. We then fit a linear model on the standardized scores, using as independent variables the loss function, dataset and NSR, as well as their 2-factor interactions with loss function. 2 To properly compare the effect of each loss function while controlling for the other factors, we compute their Estimated Marginal Means (EMM) [37], as well as 95% confidence and prediction intervals, that is, what to expect on average over multiple training runs, and what to expect of an individual training run. Fig. 3 presents the EMM standardized performance scores of all metric-based losses except . These overall results show that, in 2 The inclusion of dataset and NSR main effects does not inform the model in any way because of the standardization, but we keep them to follow the hierarchy principle of linear models.  Figure 3: Estimated Marginal Mean (standardized) score for each metric_optim. Each panel represents an evaluation metric. Small bounded segments represent 95% confidence intervals. Long unbounded segments are prediction intervals.
LambdaRank, optimizing .95 achieves the best performance across all 6 evaluation metrics and shows a consistent and statistically significant advantage over the other losses. In listwise models, our -inspired loss also achieves a statistically significant advantage over the others on all 6 evaluation metrics. These observations demonstrate the power of optimizing . To further explore the stability of such performance gain across different datasets, we show in Fig. 4 the EMM scores, but faceted by dataset. Although the general superiority of -based learning objectives is statistically significant on average, the overlapping prediction intervals indicate that it is not always necessarily the best option. In LambdaRank, the .95 loss performs best in all datasets except Epinions, where it is not statistically different from the and losses. Interestingly, we see that optimizing .8 lies in the opposite extreme and consistently achieves the worst performance. Even its prediction interval seldom overlaps   Figure 4: Estimated Marginal Mean (standardized) score for each metric_optim, by evaluation metric (rows) and dataset (columns). Small bounded segments represent 95% confidence intervals. Long unbounded segments are prediction intervals.
with that of optimizing .9 or .95. In listwise recommendation, the -inspired loss shows significant superiority over in a nearly consistent fashion, except for some notable cases like evaluating in the Epinions dataset or .95 in the Home & Kitchen dataset. It also performs significantly better than in general, but both losses yield otherwise similar performance in several cases, especially in the Epinions and Sports & Outdoors datasets. On the CiteULike dataset, however, the advantage of based models is very clear and there is even no overlap between the prediction intervals, except when evaluating .
All in all, we draw the conclusion that optimizing .95 can help achieve recommendation effectiveness at least not worse than when optimizing other informative metrics, such as and . Furthermore, the performance of optimizing on different metrics is homogeneous, which means that our -inspired listwise loss is able to help maximize scores regardless of . Moreover, according to the unreported results from our training logs, this homogeneity is not only achieved in evaluation scores, but also in the convergence process. When training with listwise -inspired losses, we manage to get the optimal values on all three values at a similar stage. In some models, the epochs to get all three optimal scores are even the same, which means that the epoch with the optimal .95 score also provides good scores on .8 and .9, and that we have the possibility to validate on only one value to satisfy different needs expressed by different values of . Hence, our listwise optimization can serve as a generic choice for rank-based recommendation. We can explain this by the nature of our listwise loss. With our transformation in Eq. (19), we do not assign different weights to items ranked at different positions. Instead, we aim at bringing all relevant items to the top of the list, and treat all positive items as equally important. This provides the model with more abundant information to train.
The observations above point to the conclusion that, although the superiority brought by optimizing for -based losses is not always significant and not fully consistent across datasets, we are still provided with a promising alternative metric to optimize in rank-based recommender systems. By optimizing -based losses, we are able to get at least comparable performance as optimizing and , with very clear benefits in many cases. In the following section, we will conduct an analysis seeking the source of performance advantage of -based listwise losses.

When to Deploy for Recommendation?
The analysis in the previous section indicated that optimizing can provide recommendation performance comparable to that of and or even better. We may trace back the source of such effectiveness and identify the best way to deploy it if we analyze the properties of the -based objective functions in the pairwise and listwise context. Similarly to the and losses, the pairwise loss in Eq. (10) guarantees strict bounds for the swap loss for each user, so that users are treated equally regardless of how many interactions they have. Contrary to this, the loss used for listwise in Eq. (19) is directly related to the predicted rank positions, which makes it have a different upper bound across users. Active users with more positive interactions are more likely to have larger loss values during training, specially for large . Such an imbalance may benefit active users, but perhaps at the cost of sacrificing the utility for inactive users. We investigated whether this effect indeed occurs in our experiments. Fig. 5  of positive items they contain in the training set. We deliberately choose to compare .95 because it achieves overall best results, and and for being the two next best losses, also with a properly bounded loss function. In addition, we only show results here for = 500%, as it is expected to amplify the aforementioned bias, if any. We find that the -inspired losses indeed perform differently under pairwise and listwise environments. In LambdaRank, where the .95 loss is also strictly bounded, we do not observe a clearly positive or negative correlation between the performance difference and the user activity level. In several cases, for active users with more items available for training, optimizing and is even more advantageous than optimizing .95. This means that active users did not benefit from the fully bounded loss, and there is no effectiveness imbalance between active and inactive users.
However, this observation does not hold in listwise models, as evidenced by the clear and significant positive correlation in all datasets except Epinions: active users do indeed benefit the most. More interestingly, this benefit for active users is not in detriment of the inactive users. As we can see in the figure, compared to optimizing and , optimizing the listwise .95 loss benefits all users on the CiteULike dataset, and achieves similar effectiveness for inactive users on the other 3 datasets. This superiority confirms our assumption that active users, whose listwise loss magnitudes are larger than for other users, can indeed get more effective recommendations due to a training process biased towards their utility, without negatively influencing the less active users. Such an insight is potentially interesting for some business applications of recommender systems, where it is beneficial to maximally serve loyal users without losing the stickiness of less active users.

CONCLUSION
Direct optimization of IR metrics has long been a hotspot in the research on ranking-based recommender systems. The intuitive and logical common practice is to build models by optimizing the same metric that will be used for evaluation. In this paper, we reported the results of an extensive experimental study aiming at acquiring new insights about the strength of the foundations behind this practice and at learning more about what metric to optimize in order to maximize recommendation effectiveness. For this purpose, we expanded the scope of metrics usually deployed to define the objective functions for LTR approaches and focused on as a promising alternative to other metrics such as , and . Experimental evidence on both pairwise and listwise frameworks show that optimizing , and generally outperforms optimizing , and that optimizing is generally no less effective than optimizing or . These findings challenge the practice to optimize and evaluate ranking-based recommender systems using the same metric. Furthermore, the new generic listwise -inspired loss proposed in this paper was shown to be able to achieve the optimal performance for different values of the user persistence parameter, without the need to specify this parameter explicitly. Optimizing this loss even significantly outperformed the direct optimization of and in some cases, showing the high potential of for developing ranking-based recommender systems. Finally, and due to the lack of a common upper bound across users, our proposed listwise loss benefits active users more than and , but without hurting the effectiveness for inactive users. This makes optimization of the proposed based listwise loss interesting for some business application cases favoring loyal users.
For future work, we will experiment with more advanced recommendation models and larger datasets to study the extent to which our conclusions and insights generalize to other settings. We will also analyze to what extent the exclusion of very inactive users affects our conclusions, especially with regards to the bounds of losses. Furthermore, it would be interesting to theoretically investigate the source of the effectiveness of even deeper.Our results show that it is a promising metric to optimize when learning to rank for recommendation, pointing to the possibility of finding even more IR metrics that could show similar potential.