{'fraction_non_alphanumeric': 0.026628140845868387, 'fraction_numerical': 0.008809948726664968, 'mean_word_length': 4.960155326692554, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Where Do We Go From Here? Guidelines For Offline Recommender Evaluation

Various studies in recent years have pointed out large issues in the offline evaluation of recommender systems, making it difficult to assess whether true progress has been made. However, there has been little research into what set of practices should serve as a starting point during experimentation. In this paper, we examine four larger issues in recommender system research regarding uncertainty estimation, generalization, hyperparameter optimization and dataset pre-processing in more detail to arrive at a set of guidelines. We present a TrainRec, a lightweight and flexible toolkit for offline training and evaluation of recommender systems that implements these guidelines. Different from other frameworks, TrainRec is a toolkit that focuses on experimentation alone, offering flexible modules that can be can be used together or in isolation. Finally, we demonstrate TrainRec's usefulness by evaluating a diverse set of twelve baselines across ten datasets. Our results show that (i) many results on smaller datasets are likely not statistically significant, (ii) there are at least three baselines that perform well on most datasets and should be considered in future experiments, and (iii) improved uncertainty quantification (via nested CV and statistical testing) rules out some reported differences between linear and neural methods. Given these results, we advocate that future research should standardize evaluation using our suggested guidelines.


INTRODUCTION
Despite growing work that tests recommender systems online, offline evaluation is still by far the most popular evaluation paradigm used in recent research publications [38]. What has been troubling is that an increasing amount of research has pointed out important issues with common protocols for offline evaluation of recommender systems [5,12,14,24] even leading some researchers to publicly call it a community-wide crisis [11]. The cumulative effect of these issues became widely visible when Dacrema et al. [12] performed a series of reproducibility experiments showing that reported gains vanished in most cases when baselines were tuned properly. In a similar vein, Rendle et al. [32] showed that proper hyperparameter selection makes traditional matrix factorization-based approaches competitive to more recent methods. Overall, these discoveries mirror findings in information retrieval over a decade prior to that in 2009 when they compared all new published models of the last 15 years and found no overall improvements [4].
Besides a lack of properly tuned baselines, there are also doubts about the statistical reliability of results. A recent survey [22] showed that 59% of RecSys papers from the previous two years used no statistical testing at all, calling statistical inference the "missing piece of RecSys experiment reliability discourse". How to set up baselines and perform statistical analysis are just two areas of many where experimenters need to make design decisions. Another area pointed out by Sun et al. [38] is dataset pre-processing, e.g., how to deal with infrequent users or items.
With all of these design choices, there has been little research on how to guide them in a principled way. In this paper, we study four common issues that arise in almost all offline experimentation setups -uncertainty estimation, generalization, hyperparameter optimization and dataset pre-processing. For each of these, we empirically examine different design choices to arrive at a set of guidelines. Even though the issues are very general, we study them in the specific and very common context of top-n recommendation from implicit feedback in this paper.
The contributions of this paper are three-fold. First, we empirically study four important issues for offline experimentation to come up with a set of informed guidelines. Second, we implement these in a lightweight and flexible toolkit called TrainRec which is designed to accelerate and support all stages during offline experimentation. TrainRec provides a set of modules that can be used independently, allowing for maximum flexibility to make integration into existing code as painless as possible. Finally, we demonstrate TrainRec's usefulness in a benchmark of a large and diverse set of baselines against a comprehensive set of commonly used recommender systems datasets. This, to the best of our knowledge, is the first consistent offline benchmark of its size comparing most recommender model classes. Our results show that many results on smaller datasets are most likely not statistically different, but there are three baselines (SLIM, MultiDAE, Ease) that perform well on most datasets, and should be considered for baselines in future experiments. Furthermore, TrainRec's improved uncertainty estimation shows that some reported differences between linear and neural models are unlikely to exist.
Overall, we hope that TrainRec gives researchers and practitioners better tools to draw reliable conclusions when trying out new algorithms. We will make the complete code to reproduce all experiments public at https://<OMITTED>.

RELATED WORK
This paper is a continuation of a longer, but recently accelerated trend in recommender systems research focused on research methodology and best practices. Around a decade ago, researchers started noticing that many recommender system papers had results that were difficult to compare due to widely differing implementations of baselines [33], diverging dataset processing choices [24], and different evaluation protocols [14]. These findings motivated a push towards open-source recommender frameworks such as MyMedi-aLite [16] or LensKit [14] which were among the first larger wave of efforts. Nowadays, there are tens of such frameworks available, with RecBole [42] and Elliot [1] being recent additions. However, having such a wide range of frameworks being available introduces new implementation differences on its own, and the bulk of replicability issues still remains unresolved. In this work, we argue that it is vital to report also how robust the outcomes of an experiment are. Without it, one cannot adequately replicate previous findings as they could have merely been to chance.
Lately, as a consequence of hugely varying evaluation protocols and baseline choices, a number of surveys have brought attention to reproducibility and replicability in the recommender community. For example, Dacrema et al. [12] discovered that recently reported state-of-the-art results were often surpassed when the set of baselines was augmented with a wider set of adequate baselines. This has also been found to be the case in session-based recommendation [28]. Since then, there has been more analysis of how consistent or diverging various elements of recommender system evaluation have been implemented in published work. Looking at the implementation of offline evaluation metrics, Tamm et al. [40] uncovered a number of troubling findings. First, there is considerable disagreement about how metrics should be implemented and Precision was the only metric that was consistently implemented. Second, only about half of all papers surveyed even specified how a certain metric was implemented. Another important factor during metric computation is what candidate set is being sampled at test time. Early work reported only smaller inconsistencies between sampling methods for precision [5], later work showed [8,25] that sampling leads to inconsistent orderings of methods and that adjustments are needed [26]. Finally, a survey by Ihemelandu and Ekstrand [22] showed that statistical testing is still missing in a lot of recently published work, making it hard to get insight into how much uncertainty was involved in the experiments.
While there has been some research on what the possible impact of different design choices for the evaluation protocol is, most studies look only at individual factors such as sampling [5,25] or metric implementation [40]. Perhaps the most comprehensive study of design choices was carried out by Cañamares et al. [9] where they looked at how the ordering of two methods (UserKNN and Matrix Factorization) on two datasets changes with different settings. As expected, choices like data splitting and score aggregation can change outcomes. On the more surprising side is the finding that even less obvious choices, e.g., changing the gain factor used in computing nDCG, can flip outcomes. Because it is impossible to resolve all these issues in a single paper, we focus on the four groups of issues in this work that we believe are most important to robust experimentation.

ISSUES IN OFFLINE EVALUATION OF RECOMMENDER SYSTEMS
In this section, we study four larger issues regarding uncertainty estimation, generalization, hyperparameter optimization and dataset processing. While there exists a much wider range of open issues, e.g., biases in the data, we chose the set below because they represent fundamental and critical building blocks of most experimentation pipelines and are thus are most likely to provide practically relevant insights.

Issue 1: Uncertainty Estimation
Quantifying the uncertainty involved in an experiment is crucial for fair comparison and replicability. To illustrate this, assume we would like to compare two baselines, UserKNN and ItemKNN on the MovieLens 1M dataset [19] and decide which one performs better. We divide the dataset up into five equal-sized folds, and use the last four of them for training (and hyperparameter optimization) and first one for testing. The settings we use in this experiment were identical to those of Section 5, but are not relevant for this exposition.
Running the two algorithms on this particular split which we call split A results in HitRate@50 scores of 0.428 for ItemKNN and 0.457 for UserKNN. Using the two-sided McNemar's test on the scores of each user, we find that UserKNN has a statistically significant higher score (p < 0.01) and are inclined to conclude that UserKNN is the best-performing method. However, if we repeat this experiment on the other four possible splits (e.g., test on fold 2), and compare those scores (Table 1), we see that the overall variance between different dataset splits is considerable and it might be unwarranted to make such a claim. In fact, as the 95% bootstrapped confidence intervals overlap ([0.432, 0.446] vs [0.441, 0.458]), we can now see that any superiority claims would be unwarranted.
We argue that the issue with the false discovery in the above experiment lies in the fact was done at the wrong level. Specifically, we only considered the uncertainty we have on the sample level, e.g., tested whether the mean scores of two models were statistically different but failed to take into account other sources of randomness that affect results which can be grouped into four categories. First, there is randomness coming from the way the data is split, and different splits can lead to considerable variance [12]. Second, many recommender algorithms use numeric methods with random components (e.g., weight initialization) that lead to different outcomes, especially for highly non-convex deep learning architectures [39]. Third, there is variance coming from hyperparameter selection [32]. Finally, as previously discussed and perhaps most used in statistical testing of recommender systems, is uncertainty at the sample level, e.g., the question of whether the population means between two labeled test sets differ [35].
Guideline 1. When reporting results, quantify the uncertainty stemming from randomness in all components, e.g., via confidence intervals. In a typical setup, there is uncertainty associated with the (i) dataset splits, (ii) hyperparameter optimization, (iii) model training, and (iv) individual predictions.

Issue 2: Generalization
Perhaps the largest practical gap lies in how recommender systems are expected to deal with previously unseen data points, for example new users, items or ratings during offline vs. online evaluation. The overwhelming majority of recommender systems are tested on the same users and items that they were also trained on, a setup also known as weak generalization. This essentially amounts to a closed-world assumption which does not reflect the reality of recommender systems at all. To demonstrate this, we examined a variety of public recommender datasets with timestamped information for each rating. Table 2 shows the number of events per day and the percentage of days with at least one event. A user was deemed a new user on the day they rated their first item. All the datasets we examined show a great amount of dynamic changes -there are at least four new users each day and we see at least one new user on more than 94% of the days. From this, we argue that any practical recommender system needs to go beyond weak generalization.
To align this terminology better with known concepts in machine learning, we use the terms transductive learning instead of weak generalization and user-inductive learning to describe the setting where test users are unknown (someimes also called strong generalization). Transductive learning is defined assumes that all test instances are known beforehand [41], whereas inductive learning treats them as unknown. As a final goal, we would like to evaluate recommender systems in a fully inductive setting with both unknown users and unknown items at test time. However, this is a very challenging scenario and there are only a few existing baselines that would support it, so we propose the intermediate goal of evaluating new systems at least in a partially inductive scenario. effective test set size score Guideline 2 (Generalization). Recommender systems should be evaluated in an inductive setting (e.g., using strong generalization) where they need to make predictions for unseen users (or items) at test time.
Looking at Table 2 again, we can see that in the datasets studied new users are seen much more often on average than new items. Because of this, we focus mainly on the user-inductive setting in this paper but note that changes to the item inventory might be more frequent in other domains (e.g., news, auctions).

User Profile Partitioning.
Related to the question of generalization is the question of how to split interactions from each user profile for training and evaluation. The most prevalent scheme according to a recent survey [38] is split-by-ratio where a certain percentage of a users' interactions go into training (e.g., 80%) and the remainder goes into validation and test.
However, there are several issues with this scheme. A smaller one is that using relative proportions with a discrete number of ratings leads to further design choices -i.e., how should one deal with users whose ratings cannot be split into the desired proportions? A far bigger issue of the split-by-ratio protocol is that it can confound results due to varying test set sizes. To illustrate this, we simulate a scenario where the recommender system produces recommendations of the same accuracy for each user and where we then gradually increase the size of each test set. More specifically, we draw a hidden set of positive items from a position-based relevance model that sets an item at rank to relevant with probability ∝ −0.05 · − . We then randomly draw test items without replacement from this hidden set and plot how a set of common metrics change with the number of test items added. Note that we have to rely on synthetic data because in real-world datasets, we cannot ensure that each user receives recommendations of the same quality. As Figure 1 shows, all metrics with the exception of Recall grow non-trivially as the test set size increases. For example, nDCG@10 grows about 4-fold when going from a test set size of one to 20. In other words, this means that a user with 20 items in their test set receives about 4 times the weight on average than a user with only one item. In essence, we argue that the split-by-ratio scheme is flawed it not gives more active users much higher weight   and thus confounds testing. To circumvent the issues above, we recommend the following: Guideline 3 (User Profiles). When partitioning user profiles, choose a hold-out protocol for validation and testing, i.e., hold items out each for validation and testing.
Of course, the larger , the fewer user profiles have sufficient ratings to be split into multiple sets. Hence, unless there is a strong reason to do otherwise, we recommend to choose the most conservative setting of = 1, i.e., adopt a leave-out-one protocol.

Issue 3: Hyperparameter Optimization
The importance of proper hyperparameter tuning has been long known in the machine learning community, but only recently received more attention in the recommender systems community [12,32]. Sun et al. [38] found that among their surveyed set of 85 recent recommender system papers, almost all of them use Grid Search even though better methods exist. To the best of our knowledge, there exists no empirical study of hyperparameter search algorithms in the recommender systems literature that could provide guidance on how to choose a set of reasonable defaults.
To this end, we ran three different hyperparameter optimization algorithms with the datasets and baselines of Section 5 with a budget of 50 evaluations each. The first method was exhaustive search through all possible parameter combinations (Grid Search), where we discretized continuous parameters to make them fit the overall budget. Second was Random Search [6] which samples points at random from the overall search space. The third method was Bayesian Optimization which samples the next hyperparameter settings based on the uncertainty it has estimated in the search space. Figure 2 shows the relative performance (best score so far over best score overall) of each hyperparameter tuning method, averaged across all datasets and baselines. Overall, we can see that Bayesian Optimization converges quickest. Grid search performs worst, both in terms of its final performance and in terms of its speed of convergence. Somewhat surprising is the competitive performance of Random Search relative to Bayesian Optimization. However, it shows less stable convergence and we recommend to only use it when parallel, independent hyperparameter optimization is needed.   To get a better picture of the number of iterations required for each baseline, Figure 3 shows the the normalized best scores found at each iteration. From it, we can see that most methods achieve 90-95% of their final performance gain within the first 40 iterations under Bayesian optimization.

Issue 4: Dataset Pre-Processing
We now turn to an issue regarding dataset handling that has received less attention in the literature. One of the most common pre-processing steps [38] is to filter out users or items that have an insufficient number of ratings. What has been less discussed is that when removing users (items) with less than interactions, the order in which these steps are carried out is critical as it can substantially change the data. To demonstrate this, we took three raw datasets, applied item and user filters with = 5 in both orders, and recorded the absolute difference in counts in Table 3. We can see that the order of filtering can vastly impact the dataset characteristics -for example, the amazon-e dataset can have about 22,000 fewer (or more) users depending on what filter was applied first. Note that interleaving those two steps via distributed processing could produce many more outcomes. To avoid this, we recommend the following: Guideline 6 (Filtering). When filtering out users or items with insufficient ratings, one should compute the -core of the bipartite user-item graph where an edge indicates a positive rating. This corresponds to filtering repeatedly until all items and users meet the desired minimum.
The big advantage of this is that the -core is unique [15], aiding reproducibility. Regarding the threshold , a reasonable starting point is to choose it at least large enough that the training validation and test portions of each user contain at least rating each.

TRAINREC: A FLEXIBLE LIBRARY FOR OFFLINE EXPERIMENTATION
Based on our guidelines, we now introduce TrainRec, a highly flexible Python library for the offline training and evaluation of recommender systems. TrainRec provides an easy and highly flexible way for practitioners and researchers to implement the above mentioned guidelines in all stages during offline experimentation. Figure 4 shows an overview of the set of modules offered by Train-Rec (light blue background). Only the model fitting itself needs to be provided; all other steps are implemented in TrainRec. Different from previous frameworks, TrainRec's purpose is not to provide a fixed experimentation workflow which often forces existing code to be re-written but to offer a set of loosely coupled modules that can be easily integrated into existing workflows. Modules can but do not have to be used together, allowing developers to opt in and out of functionality separately. Table 4 lists other common recommender frameworks which provide at least partial support for evaluation. Only TrainRec includes support for inductive generalization and uncertainty estimation.

Implementation of Guidelines
We will now discuss how each of the guidelines discussed in Section 3 is implemented.

Uncertainty Estimation.
To quantify uncertainty, TrainRec offers a procedure that combines nested cross-validation with bootstrapping. Algorithm 1 provides pseudo-code for the two steps. Intuitively, the NestedCV procedure [20] corresponds to re-running the same experiment on different partitions, each time performing a full hyperparameter search over the new validation partition. Output is a list of scores that now adequately reflects the variability from hyperparameter selection, model fitting and dataset splitting.
To make the resulting uncertainty estimates easily interpretable, we opted to compute confidence intervals via bootstrapping. Future work could examine the use of different tests, e.g., an Almost Stochastic Order test [13]. Another benefit of using this procedure is it does not rely on fixed data splits and thus the only parameter needed to replicate an experiment is the number of folds .

Generalization.
TrainRec focuses on learning and evaluation in a partially inductive setting (e.g,. either generalizing with respect to new users or items). To do so, it offers data splitting routines that divide the set of users (or items) in non-overlapping partitions (Guideline 2). A user (item) can then be further split via a hold-out protocol (Guideline 6).

Dataset Pre-Processing.
TrainRec provides an efficient implementation for computing the -core for user-item graphs (Guideline 2). In addition, it contains routines to load common datasets into a sparse matrix format.

EXPERIMENTS AND RESULTS
In this section, we demonstrate the usefulness of TrainRec by performing a comprehensive benchmark over ten datasets comparing twelve common baselines. We seek to answer the following two research questions: RQ1 What does TrainRec's uncertainty estimation reveal about the uncertainty of common methods on real-world data? RQ2 How do common baselines perform when evaluated fairly following Guidelines 1-6?

Baselines.
We considered the most commonly used baselines that support user-inductive generalization (Guideline 2). There were roughly five groups of models: (1) Unpersonalized: Random and Popularity make recommendations independent of a user in the dataset based on random scores or sorted by their item popularity. (2) Similarity-Based: Includes UserKNN, ItemKNN, and newer variants such as P3Alpha and RP3beta [31]. (3) Factorization-based: PureSVD applied a vanilla singular value decomposition to the user-item matrix; ALS implemented via weight matrix factorization and a fold-in step [21]. Note that even though popular, BPR is not a user-inductive method. (4) Deep learning-based: MultiVAE and MultiDAE [27] are two popular autoencoder-based architectures that can easily process new users.

Metrics.
Because of its simple and consistent definition, we considered HitRate@ with with = 50 as our primary metric (other values of will be reported but not optimized over). HitRate@ simply measures whether a held-out item occurred in the top or not and is equivalent to Recall@ when the test set size is one: where is the result at rank and rel( ) returns the relevance of that document.

Uncertainty Estimation.
To quantify uncertainty, we used Algorithm 1 with = 5 folds. We optimized hyperparameters inside the loop using the users from the first three folds as the new training set and the fourth one as validation. Each user profile that was not part of training was split into a fold-in, validation ( = 1) and test portion ( = 1) following Guideline 3.

Hyperparameter Search.
Hyperparameter search was done using Bayesian Optimization (Guideline 4 with a budget of 50 search iterations following Guideline 5. Hyperparameter ranges and names can be found in the supplementary material.

Datasets.
For this benchmark, we used a set of 10 publicly available and commonly used datasets, adding four more datasets to the six datasets used by the largest study we know of [38].

Dataset
Pre-Processing. We used 5-core filtering (Guideline 6) after binarizing all datasets with a threshold of ⌈ 4 5 ⌉ given a rating scale with levels except for the kuai dataset [17] for which we used the threshold proposed by the authors. Table 5 lists the ten datasets with their basic statistics. Among the datasets considered, netflix and ml100k had the most and least number of ratings respectively. Orig. sparsity describes the original sparsity level in the data and was highest for kuai which had ratings for almost 100% of all items and jester in which users rated about 50% of the items.

Results
We provide the averaged results over five different test folds in Table 6 along with their bootstrapped 95% confidence intervals (CI) for reference. Note that these results are for mainly reference and we will provide a summary graph below. Because Ease requires us to invert a matrix whose size is the number of items squared, we were unable to obtain results for the dataset with the largest set of items, yelp, running out of memory. Two datasets that stand out from the rest are kuai and jester -both were the results of attempts to create very dense or fully observed datasets [17,18]. On both of these datasets, the Popularity baseline performs very competitively. For example, on kuai, Popularity matches the performance of much more sophisticated MultiDAE method. On the jester dataset, Popularity even exceeds the performance of the typically robustly performing PureSVD baseline. Put another way, in these datasets most users like popular items (or vice versa). We will return to the discussion of the implications of item popularity in greater detail in the next section.

RQ1
: What does TrainRec's uncertainty estimation reveal about real-world experiments? Starting with the smallest dataset we had in our benchmark, the ml100k dataset, we see that its results have wide confidence intervals. For example, RP3beta has confidence intervals for HitRate@50 overlap with those of seven other methods. The slightly larger datasets lastfm and kuai have similar levels of uncertainty associated, and even the ML1M dataset that is about six times as big as ml100k still has considerable uncertainty, e.g., the CI of the top five methods all overlap. It is also worth noting that even the largest datasets still have some uncertainty associated with them, albeit much smaller ranges. However, given that papers sometimes claim superiority based on a .005 difference, those differences are unlikely to hold up when an experiment is re-run with the same or slightly changed inputs.
Note that the number of users is likely a big factor in the overall uncertainty because the individual estimates are averages over all test fold users. To see this, compare the confidence intervals of ml100k with those of jester -even though jester has a small number of items it has sufficient users to average over.
Overall, these findings imply that caution needs to be exercised -especially with small, but also with larger datasets due to the amounts of uncertainty involved. We will discuss further implications of this in the next section.

RQ2
: How do common baselines perform when following all guidelines? Since metrics have different ranges on different datasets, we computed the normalized scores for each dataset by expressing its performance relative to the best performing method on that dataset. For example, on netflix, ALS obtained a HitRate@50 of 0.424 and the best method was Ease with 0.467, resulting in a normalized HitRate@50 of 0.242/0.467 ≈ 0.906. Figure 5 shows the normalized HitRate@50 of each method averaged over all ten datasets. Clearly, random is performing worst. Next comes Popularity with a normalized HitRate@50 of 0.51meaning that it roughly reaches 50% of the total performance the best methods would have gotten. The group of middle performers are ALS, ItemKNN, UserKNN, RP3beta and MultiVAE with scores ranging from 0.89-0.93. Finally, top performing with a average normalized score of 0.97 or higher were SLIM, MultiDAE and Ease.
We also report normalized Recall@ for smaller cutoff values. Even though one is tempted to compare performance within method for each this is not a valid comparison because the normalization differs between cutoffs. However, we can still compare methods using these secondary metrics (which were not optimized over) under a fixed cutoff . When comparing the top three methods, MultiDAE appears to have lower relative recall performance at lower cutoff values ( = 10, 25). This might be due to the smoothing properties of autoencoders, but further experiments are needed to verify this hypothesis.
An interesting observation is that set of top performers consists of only one neural method and two linear models. The finding that linear models are highly competitive has also been noticed by others. For example, Jin et al. [23] also found only minute differences in performance among the SLIM, MultiDAE and Ease. They also tested variants of Ease but those failed to consistently improve over the SLIM, MultiDAE or Ease in its original formulation, again underlining how competitive they are. One difference though is that they found MultiVAE to perform slightly better than MultiDAE -but they only reported results for one run, making it hard to generalize their conclusion. The similar performance of SLIM and Ease is very likely due to their close connection -Ease essentially is the same as SLIM without an L1 sparsity constraint [37].
To answer our second research question, our large benchmark using 10 datasets and 12 baselines shows that among all baselines, SLIM, MultiDAE and Ease behaved most competitively. We recommend that those should be included in future experiments whenever possible.

DISCUSSION
We now discuss some of our findings in greater detail. When looking at the full set of results, we pointed out two potentially unwanted behaviors. First, smaller datasets had large confidence intervals associated with them. This is problematic when wanting to make superiority claims on such datasets, especially when only a single split is being used. A straightforward solution to this problem would be to drop such small datasets and only focus on larger ones. However, this may get rid of interesting dense datasets such as kuai which are mostly small in size (due to the high cost of labeling). Another potential approach could be to increase the number of outer folds -in the extreme, leave-one-user-out -so that the confidence interval will be computed over more samples. The second potential issue we uncovered was that on the two dense datasets, the Popularity baseline performed quite competitively. This finding points to a more fundamental question of what portion of the observed popularity of items is due to intrinsic factors, such as item qualities, and which portion is due to confounding factors such as perceived popularity from others [7].
In our large-scale benchmark, we found that Ease, SLIM and MultiDAE were among the best performing baselines. This finding is somewhat surprising -Ease is a simple linear model with a closed form solution whereas MultiDAE as a neural model needs to be trained via costly gradient descent. It also raises the question of what neural models are able to add beyond matching performance. We plan to study this question in more detail in future work.
To end our discussion, we would like to emphasize the list of issues we looked at is not exhaustive. In particular, offline evaluation has limitations on its own -e.g., selection biases due to interfaces or user trust [29]. However, our focus here was to provide guidance on what we believe are the most important and practical steps towards reliable offline experimentation. We would also like to emphasize that the guidelines we provide in this paper are starting points and acknowledge that there are circumstances in which there are good reasons to decide differently.

CONCLUSIONS & FUTURE WORK
In this paper, we examine four larger issues that are part of most offline evaluation protocols for recommendation and derive a set six guidelines to serve as a starting point for practitioners and researchers. To implement these, we introduce TrainRec, a Python toolkit which supports all common stages during offline experimentation. Different from other frameworks, TrainRec provides uncertainty estimation and supports evaluation in partially inductive settings.
We show how TrainRec can reveal importance insights about the robustness of offline results. In a large-scale benchmark, we compare twelve common baselines on ten datasets and find considerable uncertainty about how well a single method does on a particular dataset. While the uncertainty is particularly large for smaller datasets, there is still enough residual uncertainty in larger datasets to call into question methods that claim superiority based on smaller differences. Our benchmark also revealed that SLIM, Ease and MultiDAE were the baselines that overall perform the strongest and we recommend that those are included in future experiments. Interestingly, MultiDAE (a neural method) and Ease (a linear model) performed statistically indistinguishable on most datasets.
We hope that the guidelines in this paper along with TrainRec will lead to improved overall replicability which in turn, can accelerate research in the longer term. There are many avenues to improve and extend our guidelines and TrainRec in the future. For example, one could add other algorithms to quantify uncertainty (e.g., through parametric bootstrapping) or add support for fully inductive evaluation. Finally, we hope that this paper will enable practitioners to make more informed decisions about their evaluation setup and will spark more discourse around experimentation and evaluation of recommender systems in the future.