{'fraction_non_alphanumeric': 0.03984477588538502, 'fraction_numerical': 0.011602610587382161, 'mean_word_length': 4.601493028872544, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Contrastive Learning for Debiased Candidate Generation in Large-Scale Recommender Systems

Deep candidate generation (DCG) that narrows down the collection of relevant items from billions to hundreds via representation learning has become prevalent in industrial recommender systems. Standard approaches approximate maximum likelihood estimation (MLE) through sampling for better scalability and address the problem of DCG in a way similar to language modeling. However, live recommender systems face severe exposure bias and have a vocabulary several orders of magnitude larger than that of natural language, implying that MLE will preserve and even exacerbate the exposure bias in the long run in order to faithfully fit the observed samples. In this paper, we theoretically prove that a popular choice of contrastive loss is equivalent to reducing the exposure bias via inverse propensity weighting, which provides a new perspective for understanding the effectiveness of contrastive learning. Based on the theoretical discovery, we design CLRec, a contrastive learning method to improve DCG in terms of fairness, effectiveness and efficiency in recommender systems with extremely large candidate size. We further improve upon CLRec and propose Multi-CLRec, for accurate multi-intention aware bias reduction. Our methods have been successfully deployed in Taobao, where at least four-month online A/B tests and offline analyses demonstrate its substantial improvements, including a dramatic reduction in the Matthew effect.


INTRODUCTION
Large-scale industrial recommender systems usually adopt a multistage pipeline, where the first stage, namely candidate generation, is responsible for retrieving a few hundred relevant entities from a billion scale corpus. Deep candidate generation (DCG) [14], a paradigm that learns vector representations of the entities to enable fast k-nearest neighbor retrieval [28], has become an essential part of many live industrial systems with its enhanced expressiveness.
Typical large-scale DCG models [14,17,32,61] regard the problem of identifying the most relevant items to the users as estimating a multinomial distribution traversing over all items for each user, conditional on the user's past behaviors. Maximum likelihood estimation (MLE) is the conventional principle for training such models. Apparently, exact computation of the log likelihood, which requires computing softmax looping over a million-or even billion-scale collection of items, is computationally infeasible. Among the various sampling-based approximation strategies, sampled-softmax [6,26] usually outperforms the binary-cross-entropy based approximations such as NCE [18] and negative sampling [38] at large scale.
However, the MLE paradigm and the sampling strategies mainly stem from the language modeling community, where the primary goal is to faithfully fit the observed texts. Indeed, live recommender systems are different from natural language texts in several aspects. Firstly, the training data are collected from current undergoing systems that might be sub-optimal and biased towards popular items. In such situations, some high-quality items can be underexplored in the training data, while an algorithm trained via MLE will continue to under-estimate the relevance of the under-explored items in order to faithfully fit the observed data. Secondly, the set of items for recommendation is much larger than the vocabulary of natural languages, e.g. ≈ 100 items in our system compared to ≈ 100 words, posing great difficulties in learning a set of fairly good representations for the entire candidate set.
In this paper, we introduce CLRec, a Contrastive Learning framework for debiased DCG in RECommender systems. Contrastive learning, which constructs self-supervised tasks to improve the discriminative ability of a model, has become increasingly popular recently in the pre-training community [11,19]. We discover that the contrastive learning paradigm has a debiasing effect that can benefit recommender systems. Specifically, our central contribution is establishing the theoretical connection between contrastive learning and inverse propensity weighting, where the latter is a well-studied technique for bias reduction [25,41,48]. Our theory complements the previous studies of contrastive learning [39].
Based on the theoretical discovery, an easy-to-implement framework is developed to efficiently reduce the exposure bias of a largescale system. Our implementation maintains a fixed-size first-in first-out (FIFO) queue [19] to accumulate positive samples and their representations from the previous batches, and use the content of the queue to serve as the negative samples to be discriminated from the positive samples of the next batch. It guarantees that all items will be sampled sometime in an epoch to serve as the negative examples. More importantly, it allows us to reuse the computed results from the previous batches, e.g., saving 90% computation cost when the queue size is 10× of the batch size. As a result, we can afford to encode complex features of the negative samples even when the negative sample size, i.e., the queue size, is very large, where the rich features can improve the quality of the learned representations of the under-explored items. The queue-based framework also offers simple yet efficient implementations for complex self-supervised tasks, e.g., discriminating whether two subsequences come from the same user's behaviors [36]. We further improve upon CLRec and propose Multi-CLRec, which employs a multi-queue design for more accurate user-intention aware bias reduction.
Our methods have been fully deployed into our live system as the default choice to serve billions of page views each day. We observe that they are capable of recommending high-quality items that are largely neglected by most current ongoing systems, and consistently outperform the previous state-or-art baselines.

OUR THEORECTICAL RESULTS ON CONTRASTIVE LEARNING IN DCG
In this section, we will reveal the connection between contrastive learning and inverse propensity weighting (IPW).

Problem Formulation
Notations. Given a dataset of user clicks D = {( , , , ) : = 1, 2, . . . , , = 1, 2, . . . , }, where , = { ,1:( −1) } represents a user's clicks prior to the -th click , , and denotes the number of clicks from the user . We will drop the sub-scripts occasionally and write ( , ) in place of ( , , , ) for conciseness. We use X to refer to the set of all possible click sequences, i.e. ∈ X. Each ∈ Y represents a clicked item, which includes various types of features associated with the item, while Y is the set of all possible items. The features of could be in any form, e.g., the item's unique identifier number, embeddings or raw data of its image and text description. The number of items |Y| can easily reach 100 million.
Deep Candidate Generation. The deep candidate generation paradigm involves learning a user behavior encoder f ( ) ∈ R and an item encoder g ( ) ∈ R . The set of parameters used by each encoder is a subset of , i.e. the set of all trainable parameters in the system. It then takes {g ( )} ∈Y and builds a k-nearest-neighbor search service, e.g., Faiss [28]. As a result, given an arbitrary user behavior sequence at serving time, we can instantly retrieve the top items relevant to the user by finding the top candidate g ( ) similar to f ( ). Most implementations use inner product ( , ) = ⟨f ( ), g ( )⟩ or cosine similarity as the similarity score. The typical learning procedure fits the data following the maximum likelihood estimation (MLE) principle: . (1) The denominator of ( | ) sums over all possible items, which is infeasible in practice and thus requires approximation, e.g., via sampling. However, the observed clicks for training are from the previous version of the recommender system. The training data thus suffer from exposure bias (i.e. missing not at random [42]) and reflect the users' preference regarding the recommended items rather than all potential items. High-quality items that have few clicks in the training data will likely remain under-recommended by a new algorithm trained via the MLE paradigm.

Understanding Contrastive Learning from a Bias-Reduction Perspective
We now introduce the family of contrastive losses that we are interested in, and reveal their connection with the inverse propensity weighting (IPW) [25,41,48] techniques for bias reduction.
Sampled Softmax. The kind of contrastive loss we will investigate is similar to sampled softmax. We thus recap sampled softmax here and will show that the minor difference is crucial. There are many variants of sampeld softmax [6,26], among which the following variant is integrated by TensorFlow [1] and commonly used: where { } =1 are negative samples drawn from a pre-defined proposal distribution ( | ). Subtracting log ( | ) is necessary for it to converge to the same solution as the exact loss in Eq. (1). Most implementations assume ( | ) = ( ) and set ( ) somehow proportional to the popularity of the items to improve convergence. In practice, we would draw thousands of negative samples to pair with each positive example. Sampled softmax in general outperforms other approximations such as NCE [18] and negative sampling [38] when the vocabulary is large [14,26,33,37].
Contrastive Loss. We study the following type of contrastive loss [39,43] under a negative sampler ( | ): where { } =1 are again sampled from ( | ) for each to pair with the positive sample. It no longer optimizes the MLE loss in Eq. (1), because it misses − log ( | ) and thus does not correct the bias introduced by sampling. Many efforts on contrastive learning have been focusing on designing a well-performing proposal distribution ( | ) [9,43]. InfoNCE [39] demonstrates that this loss maximizes a lower bound of the mutual information between and if we set the proposal distribution ( | ) to be the actual data distribution data ( ), i.e. if we sample proportional to its frequency in the dataset.
Contrastive Learning and Exposure Bias Reduction. The contrastive loss as shown above in Eq. (3) has recently achieved remarkable success in various fields [11,19,39]. Nevertheless, it still remains a question why the loss is effective. We will reveal that the contrastive loss is a sampling-based approximation of an inverse propensity weighted (IPW) loss. The IPW loss 1 has the form: where ( | ) should be the propensity score function, which represents the impression (or exposure) distribution, i.e., the probability that item is recommended to user in the previous system where we collect the training data D. The idea of IPW is to model missing-not-at-random via the propensity scores in order to correct the exposure bias. We can prove that the IPW loss is optimizing ( | ) to capture the oracle user preference even when there 1 Our IPW loss is different from the previous works on debiased recommenders [42,52]. We focus on multinomial propensities, i.e. whether an item is selected and recommended by a recommender out of all possible items. The previous works consider bernoulli propensities related with users' attention, i.e. whether a user notices a recommended item or not, and mostly deal with the position bias in ranking. We give the derivation of this multinomial IPW loss in the Appendix. exists exposure bias. A standard implementation of the IPW loss has two steps, where the first step is to use a separate model to serve as ( | ) and optimize it by fitting the exposure history according to the MLE principle, while the second step is to optimize ( | ) according to Eq. (4). However, the two-stage pipeline of IPW, as well as the numerical instability and variance [42] brought by 1 ( | ) , makes IPW less efficient for large-scale production systems.
Fortunately, we can prove that the contrastive loss Eq. (3) is in principle optimizing the same objective as Eq. (4). And in Subsection 3 we provide simple implementations that do not require two separate steps and can avoid the instability brought by the division of ( | ). Our key theoretical result is as follows: Theorem 1. The optimal solutions of the contrastive loss (Eq. 3) and the IPW loss (Eq. 4) both minimize the KL divergence from ( | ) to ( | ) = data ( | )/ ( | ) what is the frequency of apprearing in D given context .
Proof. See the Appendix. □ Remark 1. The implication of Theorem 1 is that the contrastive loss (Eq. 3) can approximately reduce the exposure bias if we set the proposal distribution ( | ) to be the propensity score, i.e. the probability that the old systems deliver item to user when we were collecting the training data D.

QUEUE-BASED CONTRASTIVE LEARNING FOR BIAS REDUCTION
We propose two efficient implementations of contrastive losses for bias reduction in large-scale DCG: (1) We will first present CLRec (see Figure 1), which draws negative samples from a queue that roughly follows the non-personalized propensity score ( ), i.e., the probability of item being recommended by the old system under which we collect the training data, for reducing the Matthew effect.
(2) We then improve upon CLRec and propose Multi-CLRec (see Figure 2), which involves multiple disentangled queues corresponding to different user intents and performs bias reduction based on the more accurate propensity score ( | user 's intent). Our approaches, unlike the IPW methods that require introducing an extra model for estimating the propensity score ( | ), are free from the variance brought by the extra model and do not need to introduce a costly, extra stage for training the extra model.

CLRec: Queue-based Contrastive Learning for Reducing the Matthew Effect
The exact propensity score ( | ) is challenging to estimate, because industrial systems involve many complicated modules and the data are sparse. CLRec thus use ( ) in place of ( | ), i.e. assuming ( | ) ≈ ( ), to ease estimation and avoid small propensities. Secondly, ( ) (i.e. the exposure probability that item is recommended to someone) has a high correlation with ( ), i.e. the click probability that item is being recommended and clicked by someone, because the existing system will mainly recommend items that have a high click-through rate if the system is already highly optimized. We thus further replace the impression distribution ( ) with the click distribution ( ), i.e. assuming ( ) ≈ ( ), to ease implementation and save computation since impression data is larger than click data by an order of magnitude.

Drawbacks of Performing Explicit
Sampling. Our analysis above shows that, by simply drawing negative samples according to ( ), the contrastive loss (Eq. 3) can alleviate the selection bias related with the items' impression counts in the history. However, sampling will still incur non-negligible overheads, e.g. communication costs, in a distributed environment [45]. Sampling also cannot guarantee that every item will be sampled in an epoch. We thus adopt a queue-based design [19] that avoids explicitly performing sampling, as shown in Figure 1b and Figure 1c.
Designs of Queue-based Contrastive Learning. We maintain a firstin first-out (FIFO) queue Q, which has a fixed capacity and can store |Q| examples. Given a new batch to process, we first enqueue the positive examples (or their representations g ( )) encountered in the present batch into Q. We then use the examples stored in the queue to serve as { } ∪ { } =1 (the positive example and negative examples) when computing the denominator of the contrastive loss (Eq. 3) for the present batch. In a distributed setting, each worker maintains its own queue locally to avoid communication costs. When the queue size |Q| is equal to the batch size, our implementation is then equivalent to sampling negative examples from the present batch [12,21,43] (see Figure 1a). In general, we need thousands of negative samples to achieve satisfying performance. We thus use a large queue size, but with a small batch size to save memory, e.g. batch size = 256 and queue size = 2, 560.
Queue-based Caching and its Efficiency in Encoding Complex Features. With the implementation that stores the encoded results g ( ) in the queue (see Figure 1c), we can no longer back-propagate through the negative examples from the previous batches, though we can still back-propagate through the negative examples from the present batch. As a result, we find that the total training steps required for convergence mildly increase. However, since each training step will take much less time to compute (and less communication costs in a distributed training setting), the total running time can still be greatly reduced if the features of the negative items are expensive to encode, e.g. if the features contain raw images, texts, or even structured data such as a knowledge graph.

Multi-CLRec: Multiple Disentangled
Queues for Intent-Aware Bias Reduction There are tens of thousands of categories of items in our system, many of which are long-tail. As a result, approximating ( | ) with ( | the category in which user is currently interested) can still incur non-negligible variance. Multi-CLRec thus aims to cluster the categories into intentions and simultaneously learn a de-biased model with negative samples that roughly follow the smoothed propensity scores ( | user 's current intention is ℎ), ℎ = 1, . . . , . In particular, Multi-CLRec uses queues {Q ℎ } ℎ=1 corresponding to the intentions. We use = 64 in our system.
Disentangled Encoding with Prototype-based Routing. Our model (see Figure 2) borrows the idea of intention disentanglement [35], which has a built-in routing mechanism for clustering. Let = { 1:( −1) } be the sequence of items clicked by user , and the user's ground-truth next click to predict is . Let c ∈ R be the embedding corresponding to item 's category ID. The embedding of item is generated by the item encoder g : Y → R as follows: where MLP 1 (·) represents a multilayer perceptron (MLP) whose input is the concatenation of item 's features. We then use another MLP to model the importance of each clicked item { } −1 =1 : time gap between clicking item and item ; user's dwell time on item ; . . . ]) and ′ ∈ R.
(6) Our model then uses a set of trainable parameters ℎ ∈ R , ℎ = 1, 2, . . . , , to represent intention prototypes [35], based on which each item is being routed into intention ℎ with probability ℎ | : The temperature hyper-parameter is set to = 0.07 following previous work [19], and ∥ · ∥ stands for the 2-norm of a vector. Based on the importance weights and the routing results, we obtain  Figure 2: Multi-CLRec, a multi-queue implementation of contrastive learning for bias reduction in DCG.
intention vectors {z ℎ } ℎ=1 for the user: where ℎ ∈ R , ℎ = 1, 2, . . . , , are trainable parameters that represent the bias vectors of the intentions, which are shared by all users. However, not all of the intention vectors would match the user's current interest, since it is possible to have −1 =1 ℎ | · → 0 for some ℎ. We thus use the following attention module to select the intention vector that is most likely to match the user's interest, and set the the user encoder f (·)'s output to be the selected vector z ℎ * : (9) We approximate the gradient of argmax via straight-through [5].
We then use ℎ instead of ℎ * when computing the loss (see Equation 13). When serving online, we can select the top-(1 ≤ ≤ ) vectors from {z ℎ } ℎ=1 with maximum values of ℎ for item retrieval, which is useful if a diverse recommendation list is preferred.
Auxiliary Losses for Enhancing Interpretability. The encoder architecture alone cannot guarantee an interpretable routing mechanism in our large-scale setting. We thus introduce three auxiliary losses to enhance the interpretability 2 . The first is to encourage ℎ | to be polarized, i.e., ℎ + | → 1 and ℎ ′ | → 0 for ℎ ′ ≠ ℎ + : The second auxiliary loss is for pushing the correct intention vector z ℎ + closer to the target item while making the incorrect 2 Empirical results that demonstrate the enhanced interpretability are provided in the Appendix. intention vectors {z ℎ ′ } ℎ ′ :ℎ ′ ≠ℎ + , far away from the target item: (11) Our third auxiliary loss is for balancing the intentions, such that each is responsible for a roughly equal number of items: where E B means that the expectation is computed based on the items encountered in the present mini-batch B.
Multi-Queue Contrastive Loss. The main loss to optimize is: where cos( , ) = ⟨a, b⟩/(∥a∥ · ∥b∥). Here Q 0 is the main queue, as described in Subsection 3.1, and Q ℎ + is the ℎ + -th secondary queue (one of the secondary queues) that stores negative samples associated with intention ℎ + = arg max ℎ ℎ | . We update the 1+ queues prior to computing the loss at each iteration as follows: (1) The positive sample is enqueued into the main queue Q 0 prior to computing the loss. (2) An item − just dequeued from the main queue Q 0 is enqueued into the secondary queue and c − is the category embedding of item − . (3) Items dequeued from the secondary queues {Q ℎ } ℎ=1 are discarded. We minimize the total loss L cl + 1 L aux,1 + 2 L aux,2 + 3 L aux,3 . We simply set 1 = 2 = 3 = 1, because we find that our method is robust with respective to a wide range of coefficients.
Remark 2. We are implicitly performing bias reduction based on a propensity score proportional to ( | user 's intention is ℎ) + · ( ) for a certain smoothing factor > 0, by using the negative samples from a secondary queue Q ℎ and the main queue Q 0 .

EXPERIMENT 4.1 Experiments in Production Envrionments
The online experiments have lasted for at least four months, and our algorithm serves several scenarios with different traffic volumes. Details can be found in the Appendix. The total number of items is around 100 million. We use the queue-based implementation without caching in this subsection, and we will explore settings where encoding is expensive and requires caching in Subsection 4.2.2.

The Debiasing Effect in Large-Scale Production Environments.
To examine the debiasing effects of CLRec, we first conduct offline experiments and compare CLRec with sampled softmax. We report  the aggregated diversity [2] in Table 1, and the distributions of the recommended items resulted from the different losses in Figure 3. Table 1 shows that CLRec has an over 2× improvement on aggregated diversity. We see from Figure 3 that, sampled softmax tends to faithfully fit the distribution of the training data. CLRec, however, learns a relatively different distribution, which shifts towards the under-explored items and alleviates the Matthew effect.
Bias reduction not only leads to a fairer system, but also contributes to a significant improvement regarding the online performance. In Table 2, we compare CLRec with other sampling alternatives using the same implementation of the encoders. Details of the alternative methods are in the Appendix. We observe that negative sampling [38], including its variant [56] that makes the instances in a batch share the same large set of negative samples, does not perform well in our settings. CLRec's improvement over sampledsoftmax [6,26] w.r.t. the offline metric HitRate@50 is negligible. However, CLRec achieves significant improvement regarding the click-through rate (CTR) online, which indicates that there exists discrepancy between the offline metric and the online performance.
In Table 4, we compare Multi-CLRec against CLRec. We deliberately evaluate the methods in the setting where the training and the test data have different user distributions. Multi-CLRec is more robust compared to CLRec and achieves higher performance.

At Least Four
Months' Large-Scale A/B testing. CLRec has been fully depolyed into several heavy-traffic scenarios since Feb 2020, after the initial proof-of-concept ablation studies shown in Table 2. Table 3 shows our main online results conducted in these heavy-traffic scenarios, with billions of page views each day. During ≈ 280 ≈ 500 the at least four months' A/B testing, CLRec has been consistently outperforming MIND [32], the previous state-or-art baseline, in terms of the fairness metrics such as aggregated diversity and average popularity index, as well as the user engagement metrics such as click-through rate and average dwell time. Compared with MIND [32], which is the previous state-of-art DCG baseline deployed in the system, CLRec tends to recommend items with a lower popularity index while being more attractive to the users. This proves CLRec's ability of identifying high-quality items that are rarely explored by the previous systems. We further achieve a +2% relative improvement in the total click number on our platform after ensembling CLRec and MIND, compared to using MIND alone.
In mid-2020, we upgrade CLRec to Multi-CLRec, and Multi-CLRec brings a 3% relative improvement in terms of total clicks on our platform while maintaining a comparable click-through rate. Table 5 compares CLRec and sampledsoftmax in terms of training speed and the network traffic required in a distributed setting. CLRec's queue-based implementation is much more efficient than the methods that perform explicit sampling, since CLRec reuses the result computed for a positive sample shortly later when the sample is serving as a negative sample. The version of sampled-softmax that encodes features for the negative items is from [62]. This proof-of-concept experiment only uses a few features of the items, and we can deduce that the improvement regarding efficiency will be much more substantial with more complex features. Table 6 shows that encoding features for the negative samples is beneficial, which justifies the efforts spent on efficiency. Table 7: Task u2i is the regular task where is a sequence of clicks and is the next click to be predicted. Task u2u [36] adds an auxiliary loss where and are both sequences from the same user (before and after a sampled timestamp), which is co-trained with task u2i. HR1@50 and Recall5@50 represent the HitRate and Recall truncated at 50 when predicting the next one and five clicks, respectively.

Complex Pretext
Task that Requires the Cached Implementation. We now demonstrate that CLRec with a queue that caches the computed results can enable more complex pretext tasks that may improve the quality of the learned representations. We consider an auxiliary task where and are both sequences from the same user (before and after a sampled timestamp). The goal is to identify the correct sequence that belongs to the same user that produces . This auxiliary task is expensive to implement with sampled-softmax, since the negative samples are sequences and are thus expensive to encode. Fortunately, cached CLRec can implement it efficiently. Table 7 demonstrates that the auxiliary task can improve an algorithm's ability to make long-term prediction. MoCo [19] proposes momentum update for stabilizing the training loss. We however observe no additional gain with MoCo. Table 9: Results on public benchmarks preprocessed by HPMN [40]. We follow HPMN [40] and use AUC as the metric. The clicks in the training and the test set of each benchmark happen before and after a certain timestamp, respectively. There hence exists a distribution drift. The users also have diverse interests, i.e., the probability of a user's two consecutive clicks belonging to the same category is low.

Reproducible Experiments on Public Data
In addition to the experiments on large-scale data, we also conduct quantitative experiments on several public benchmarks. The quantitative results are listed in Table 8 and Table 9, respectively. We also conduct the same analyses as in Subsection 4.1.1 to demonstrate the debiasing effects of CLRec on the public benchmarks. We sample a sequence from each user in the training set as the input to the recommenders. Each recommender then retrieves ten items for each sequence. The results are in Table 10 and Figure 4. Table 10 reports aggregate diversity of the items retrieved by the algorithms, as well as the aggregate diversity of the groundtruth next clicks. CLRec retrieves a more diverse set of items than sampled softmax, i.e., much more items will have a chance to be recommended after we reduce the bias in the training data. Figure 4 visualizes the distributions of the items retrieved by each algorithm, as well as the training data's distribution. Sample softmax tends to faithfully fit the training data's distribution, while CLRec tends to explore the previously under-explored items.

RELATED WORK
Deep Candidate Generation. Deep candidate generation methods are widely deployed in industrial systems, e.g., YouTube [10,14,27], Taobao [32,35,59,61], and Pinterest [56]. The existing methods explicitly sample negative examples from a pre-defined proposal distribution [6,26,38]. The proposal distribution not only affects convergence, but also has a significant impact on the performance [9]. Empirically the number of the negative samples need to be large, e.g. a few thousand ones for pairing with a positive example. Consequently, it is computationally expensive to incorporate rich features for the negative samples. The existing systems hence usually choose to not encode features for the negative examples except for simple features such as item IDs [14], even though rich features for the negative samples are demonstrated to be beneficial [4]. CLRec achieves great efficiency when encoding rich features for the negative samples by caching the computed results.
Bias Reduction and Fairness in Recommender Systems. Recommendation algorithms that directly fits the training data will suffer from selection bias due to the missing-not-at-random phenomenon [42], where the previous recommendation algorithms affect the training data collected. The topic of reducing the bias in training and evaluating recommender systems has been explored before [3,10,20,42,44,52,54,57,63]. However, these existing works mostly focus on small-scale offline settings, and rely on techniques less efficient for large-scale DCG. For example, most of them involve an extra stage to train a propensity score estimator. Dividing the propensity scores can also lead to high variance when small propensities exist [42]. Correcting the bias helps improve P-fairness [8], i.e. fairness towards the under-recommended products [7].
Contrastive Learning. Contrastive learning, which aims to learn high-quality representations via self-supervised pretext tasks, recently achieves remarkable success in various domains, e.g., speech processing [39], computer vision [11,19,22], graph data [50], and compositional environments [30]. The contrastive loss we investigate in this paper is a generalization of the InfoNCE loss [39]. InfoNCE is previously understood as a bound of the mutual information between two variables [39]. Our work provides a new perspective on the effectiveness of the contrastive loss, by illustrating its connection with inverse propensity weighting [25,41,48].

CONCLUSION
We established in theory the connection between contrastive learning and IPW, based on which we proposed CLRec and Multi-CLRec for efficient and effective bias reduction in large-scale DCG.

A EXPERIMENTAL SETTINGS
Data and Hyper-parameters. We use the data collected from the last four days on our platform as the training data and use the click data recorded in the following day as the test set. We limit the minimum length of a valid sequence to be five in the training set, because requiring the model to predict the next item of an unusually short sequence can easily introduce great noise which harms the recommendation performance. We use seven categorical features of the items, including item ID, coarse-grained and fine-grained category IDs, seller ID, seller category, brand ID, and gender preference, for our live experiments. We use the queue-based implementation without the caching mechanism on the regular tasks where the negative items consist of the seven categorical features, while using the queue-based implementation that caches the processed results when solving the complex pretext tasks due to the high costs of encoding the complex negative examples. The number of negative samples is 2,560 while the batch size is 256. Multi-CLRec use = 16 on public data and = 64 on our large-scale data. We train the model for one epoch in total on our large-scale data.
Training Environment. The training data contain four billion sequences of user behaviors. We train the model in a distributed TensorFlow cluster that consists of 140 workers and 10 parameter servers. Each worker is equipped with a GPU, which is either GTX 1080 Ti or Tesla P100. Each parameter server is allocated 32GB memory while each worker has 16G memory. Training CLRec and Multi-CLRec on our internal four-day data for one epoch costs less than eight hours for task u2i without caching and less than twelve hours for task u2u with caching.
Online Serving. We pre-compute offline the item representations via the item encoder. We then store and index the item representations using an online vector-based kNN service for fast retrieval of top relevant items at serving time. At serving time, our user encoder will infer a user vector f ( ) as the request made by user arrives and pass the user vector to the kNN service to find a few hundred relevant items . The latency for dealing with the query is around 10 ms. The set of relevant items is then send to be scored by a separate deep ranking model that costs much more computational power and is responsible for constructing the final recommendation list by selecting the top relevant items from .
Evaluation Metrics. We report the performance in terms of the following offline and online metrics: • HitRate@50 (HR@50) is an offline metric. We sample a subset of users and compute the metric based on the users' click sequences to measure an algorithm's offline performance. We pick the latest one click of a user as the label to predict, and use the user' sequence prior to to serve as the input . We run the algorithm under evaluation to retrieve top 50 items for each user via kNN searching. If the ground-truth next one click is in the retrieved 50 items, the algorithm receives a score 1 for this user, or score 0 otherwise. The HitRate@50 of the algorithm is then the average score received by the algorithm.
• The click-trough rate (CTR) of an algorithm is the percentage of recommendations made by the algorithm that are finally clicked by the users. • The average dwell time is the total time spent by the users on reading the details of the items clicked by them, divided by the total number of clicks on our platform. • The aggregate diversity [2], measured on a sampled subset of users for testing, is the number of distinct items recommended to the subset of users. • The popularity index of the items recommended by a candidate generation algorithm is defined as Here data ( ) is the number of times our system recommends item to the users in the past few weeks before we deploy CLRec into the system, and E [ data ( )] = ∈Y data ( )· (algorithm recommends item ) measures how much an algorithm prefers recommending the items that are already popular. The algorithm ′ that has the largest value of E ′ [ data ( )] in our system is a simple candidate generation method that does not learn vector representations, which is used as a fallback to handle the cases when the deep candidate generation methods, i.e. MIND and CLRec, fail to return the results within a time limit.

B IMPLEMENTATION DETAILS B.1 Single-Queue CLRec's Encoders
The encoders of the multi-queue implementation Multi-CLRec is described in Subsection 3.2. However, the single-queue CLRec uses a different set of user and item encoders, which is due to the fact that our large-scale experiments related with the single-queue CLRec were conducted before we invented Multi-CLRec. We thus describe in this subsection the single-queue CLRec's encoders.
B.1.1 Item Encoder. In the single-queue CLRec, an item's embedding is the sum of several parts, i.e. a base item embedding corresponding to its ID umber, the concatenation of the item features' embeddings, a time bucket embedding, as well as a backward positional embedding. The time bucket embedding and the positional embedding are only added when the item embedding is to be used by the user encoder f ( ) for encoding a past click made by user . The output of the item encoder g ( ) for item is the sum of only the base item embedding and the item features' embeddings, excluding the time and positional embeddings. Notably, Multi-CLRec also use the time and positional embeddings described here for estimating each click's importance weight.
Time Bucket Embedding. To encode time intervals between consecutive behaviors, we use the embeddings of bucketized time intervals, which are used by previous work [58]. To be specific, given the current timestamp when the user query arrives and the timestamp ′ when a click was recorded, the time interval − ′ is discretized and put into a bucket that corresponds to a specific range. Each bucket has its own vector representation, i.e. bucket embedding.
Backward Positional Embedding. We also find that the positional embedding has extra positive impacts even when the time bucket embedding is already in use. There are two ways to add positional embeddings. The forward way is to view the latest click as if it is at the anchor position, i.e. position zero, while the earliest click of the user would then be at a varying position depending on the user's sequence length. The backward way is, on the other hand, to view the earliest click as if it is at position zero, and the latest click would then be at a varying position when the sequence length varies. We find that the backward version of the positional embedding brings extra improvement even after the bucket embedding is used, while the forward positional embedding does not. We thus adopt the backward positional embedding. The time bucket embedding has already highlighted the recent behaviors, and the backward positional embedding may be behaving like the backward direction in a bi-directional LSTM for capturing long-range dependencies.
Multi-Head Attention. Multi-head attention [49] achieves great success in various NLP tasks and is adopted by sequential recommendation models [29,58]. We simplify the multi-head attention [34] for efficiency's sake and implement the following module: where X ∈ R ( −1)× are the − 1 clicks' embeddings. MLP is a two-layer feed forward neural network whose output is of shape R ( −1)× , representing the sets of attention scores for the attention heads, respectively. The softmax is performed along the last axis of its input, hence the -1. The output of the above module is a set of -dimensional embeddings, namely head embeddings.
Weighted Heads Aggregation (WHA). After the simplified multihead attention module, there are head embeddings {z ℎ } ℎ=1 . We use a WHA module to aggregate the head embeddings into one single user embedding. WHA first produces a global feature vector m ∈ R , based on which another weighted attention is performed to produce the final user embedding f ( ) ∈ R for user : where A ∈ R × is a learnable parameter. We find WHA to be more effective than mean-pooling or concatenation.

B.2 Normalization and Initialization
WHA and contrastive learning both seem to be sensitive to the similarity metrics used. We thus 2-normalize the head embeddings before sending them into WHA and normalize the output of WHA as well, which empirically improves training convergence and recommendation diversity. We use the cosine similarity with temperature > 0 for contrastive learning, i.e. ( , ) = ⟨f ( ),g ( ) ⟩ ∥f ( ) ∥ ∥g ( ) ∥ , where = 0.07 following previous work [19].
Some efforts [35,51,53] have been spent on analyzing the cosine similarity related with 2-normalization as well as the positive effects brought by 2-normalization. According to [53] which assumes a normal distribution, cosine softmax will force the embeddings to form more compact clusters, which makes kNN search more easy.
The embeddings should use normal initialization [16] instead of uniform initialization if 2-normalization is in use, so that the initial embeddings are uniformly spread over the hyper-sphere, which prevents the loss from oscillating at the beginning of training.

B.3 Loss Functions and Sampling Strategies
Negative sampling [38]. We sample examples to pair with each positive click from a proposal distribution ( ). The proposal distribution ( ) is proportional to the item's degree, i.e., degree( ) 0.75 as recommended in [38]. We use a distributed version of the sampling strategy based on the alias method [45], which is provided by AliGraph [62]. We tune from 8 to 2, 560 and report the best. Sampled softmax [26]. We use the sampled softmax implemented in Tensorflow [1], which subtracts the correction term to make the sampled loss approximately optimize the same objective as the full loss. We set the number of negative examples used by sample softmax per batch to be the same as the queue size of CLRec.
CLRec & Multi-CLRec. The batch size is 256. The queue size of CLRec is 2,560. The queue size of Multi-CLRec is 1,280, leading to 2,560 negative items in total as it draws samples from both a main queue and an additional secondary queue.

B.4 Complex Pretext Tasks
In task u2u [36], and are both sequences from the same user, before and after a sampled timestamp, respectively. Intuitively, pre-training a recommender to solve task u2u may improve a recommender's ability to make long-term prediction. We co-train the same encoders to solve both task u2u and the original u2i task where and are a sequence and the sequence's next click. Task u2i is still required, since we ultimately need the recommender to recommmend items rather than sequences at serving time. The batch size is 256. We use one separate queue of size 2,560 for each task. We limit the sequence length of in task u2u to be less than ten. We also investigate whether Momentum Contrast (MoCo) [19] may improve performance, and vary the momentum parameter in the range of {0.9, 0.99, 0.999, 0.9999} and report the best result.

C ADDITIONAL EMPIRICAL RESULTS
In this section, we provide additional empirical results that demonstrate the interpretability of Multi-CLRec and justify some designs of our encoders. (1) Top items retrived by vector 1.

C.1 Results about Multi-CLRec's Encoders
In Figure 5, we demonstrate that the top-vectors selected by Multi-CLRec's user encoder do represent a user's different intentions, which is mostly due to the auxiliary losses for intention disentanglement. We note that the case shown here is not cherry-picked, but a randomly sampled case.

C.2 Results about CLRec's Encoders
In this subsection, we provide some interesting empirical findings that motivates the design of our encoders when developing the single-vector CLRec, even though the findings are not strictly related to contrastive learning or bias reduction. Each of the following experiments should be taken as the analysis for each component individually, which are conducted during different time periods. Considering the costs of deployment into a live system, we deploy a model online only if it has at least similar performance with the previous baselines, and mark a method's online performance as "-" in the table if it is not deployed. Table 11 then demonstrate the usefulness of combining time bucket embedding and backward positional embedding.
We then illustrate in Table 12that the encoders can benefit from the combination of weighted head aggregation (WHA) and the cosine similarity, where the results show that cosine similarity is indeed needed for reaching a higher HitRate.
We also find that combining WHA and cosine can endow a single-vector user encoder with the ability to generate a diversified recommendation list that reflects the different intents behind a user's behavior sequence (see Figure 6). Without WHA, we observe that the retrieved item set can be easily dominated by one single intent, showing limited diversity.
The previous methods, especially those use dot product instead of cosine similarity, easily lead to a much skewer interest distribution of the top-k retrieved items. These previous methods tend to under-explore items from the less popular interests. Many have also noticed this issue and proposed their solutions, such as using multiple interest vectors [32,35] or multiple interest sub-models [61]. Our result (Figure 6), however, indicates that a single-vector model can sometimes be sufficient for generating a diverse recommendation list.

D THEORECTICAL ANALYSES AND PROOFS
In this section, we provide the derivation of the IPW loss for a multinomial policy, along with the proof of our theorem.
(b) Items retrieved by the concatenation method.
(c) Items retrieved by the WHA + cosine method. Figure 6: It is possible for a single-vector user encoder to generate a diverse recommendation list, as long as the model is properly designed, e.g., if the model combines the weighted head aggregation (WHA) strategy with the cosine similarity. Nevertheless, the single-vector CLRec is not as controllable and reliable as the multi-vector Multi-CLRec when it comes to preserving the multiple intentions in a balanced manner.

D.1 Derivation of Inverse Propensity Weighting for Multinomial Policies
In this subsection, we derive the IPW loss for DCG, where we represent the recommendation policy as a multinomial distribution. We assume that the recommender system is a sequential recommender system where a user will receive only one recommendation when the user's state becomes . However, we note that it is easy to verify that our conclusions still hold when user state receives recommendations, where ≥ 1. We will also focus on a single user state for conciseness.
D.1.1 Training with Unbiased Data. Ideally, we should collect the training data that do not suffer from selection bias. This is equivalent to collect training data under a random recommendation policy uni that always recommends a random item in a uniform way, i.e., the random policy should not be aware of the value of and .
Let the oracle user preference be (click = 1| , ), which is a bernoulli distribution and represents how likely user will click item if we recommend item to the user. Assuming that user clicks one item recommended by the random policy uni , the click data collected under uni will be: which is the oracle click data distribution for unbiased training, since the uniform random policy uni does not introduce any selection bias. Note that uni ( | ) is a multinomial distribution rather than a multivariate bernoulli distribution.
To learn an ubiased recommendation policy, we need to optimize ( | ) on the click data collected under the uniform random policy. In other words, the unbiased loss function should be: ( | ) = − ∑︁ uni ( | ) log ( | ).
D.1.2 Training with Biased Data: Inverse Propensity Weighting. However, random exploration is expensive in a real-world recommender system. In practice, the click data D for training a new policy is collected under a biased old recommendation policy . Let ( | ) be the probability that the old policy recommends item to a user in state . Note that ( | ) is a multinomial distribution, i.e. ′ ( ′ | ) = 1. The generating process of the biased dataset D is as follows. Policy makes a recommendatin, i.e. draws a one-hot impression vector from the multinomial distribution ( | ), whose th element = 1 if the recommendation is item , and = 0 otherwise. On the other hand, user is associated with a multi-hot vector representing the user's preference regarding all the items, whose th element is drawn from the bernoulli distribution (click = 1 | , ), = 1, 2, 3, ..., |Y|. Here = 1 if the user will click item after being recommeded item , = 0 otherwise. A naïve estimator that does not take selection bias into account typically learns a new biased policy ( | ) by minimizing the following biased loss: Since and are independent, the expectation of the above loss is The naïve estimator is thus not optimizing the ideal loss as described in Eq. (15). Now let us analyze the IPW estimator, which minimizes the following loss: Note that uni ( | ) ∝ (click = 1 | , ) according to Eq. (14). We thus have: We can see that, if we set ( | ) to ( | ), the IPW estimator is then an unbiased estimation that minimizes the oracle unbiased loss Eq. (15).

D.1.3 Bernoulli Propensities vs. Multinomial Propensities.
We note that our fomulation is different from the existing literature on debiasing a recommender [3,10,42,44,52,54]. Most of the existing methods assume that the recommendations are drawn from a multivariate bernoulli policy, i.e., follows an independent bernoulli distribution (recommend = 1 | , ) [42,52]. Here ∈ {0, 1} denotes whether item is recommended by the policy. We take a different approach and instead assume that is drawn from a multinomial policy ( | ) rather than a multivariate bernoulli one.
In other words, the difference lies in whether we should design a candidate generation policy as a multinomial model or as a multivariate bernoulli model. We choose to design the policy as a multinomial one, due to the following reasons: • The multinomial formulation brings superior performance.
There are many empirical results that report better performance with a multinomial candidate generation method than a multivariate bernoulli one, especially with a large set of items for recommendation [14,26,32,33]. Our empirical results in Table 2, where the negative sampling baselines follow the multivariate bernoulli formulation, also verify this finding. Similarly, the natural language processing (NLP) community also report better results with a multinomial classifier, especially when the number of classes increases [37]. • The multinomial formulation is more consistent with the realworld production environments. The multivariate bernoulli formulation implicitly assumes that the number of recommendations requested by a user can be modeled as part of the recommendation policy , e.g., the expectation of the number of recommendations is (recommend = 1 | , ) with the multivariate bernoulli formulation. However, this assumption does not hold in many recommender systems, where the number of recommendations received by a user is decided by the user rather than the system. For example, the user can request more recommendations by scrolling down the page or stop receiving any new recommendation by leaving the page. On the contrary, the multinomial policy ( | ) does not attempt to model the number of recommendations requested by a user. Rather, the multinomial policy ( | ) only models which one item it should recommend if the user explicitly requests the system to make one recommendation.