{'fraction_non_alphanumeric': 0.03008334948633456, 'fraction_numerical': 0.010312075983717774, 'mean_word_length': 4.791647956892681, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
UserBERT: Contrastive User Model Pre-training

User modeling is critical for personalized web applications. Existing user modeling methods usually train user models from user behaviors with task-specific labeled data. However, labeled data in a target task may be insufficient for training accurate user models. Fortunately, there are usually rich unlabeled user behavior data which encode rich information of user characteristics and interests. Thus, pre-training user models on unlabeled user behavior data has the potential to improve user modeling for many downstream tasks. In this paper, we propose a contrastive user model pre-training method named UserBERT. Two self-supervision tasks are incorporated in UserBERT for user model pre-training on unlabeled user behavior data to empower user modeling. The first one is masked behavior prediction, which aims to model the relatedness between user behaviors. The second one is behavior sequence matching, which aims to capture the inherent user interests that are consistent in different periods. In addition, we propose a medium-hard negative sampling framework to select informative negative samples for better contrastive pre-training. We maintain a synchronously updated candidate behavior pool and an asynchronously updated candidate behavior sequence pool to select the locally hardest negative behaviors and behavior sequences in an efficient way. Extensive experiments on two real-world datasets in different tasks show that UserBERT can effectively improve various user models.


Introduction
User modeling is a core technique for various personalized web applications such as personalized news (Okura et al., 2017) and item recommendation (Sun et al., 2019). Existing user modeling methods usually model users from their behaviors (Zhou et al., 2019). For example, Wu et al. (2019d) proposed a hierarchical user representation model for user demographic prediction that models users from their search queries.  proposed to use an attentive multi-view learning framework for native Ad click-through rate prediction, which models users based on their search query and webpage browsing behaviors. These methods usually require a large amount of labeled data for training accurate user models (Yuan et al., 2020). However, in many scenarios labeled training data is insufficient, and it is often expensive and time-consuming to collect or annotate (Wu et al., 2019d). Fortunately, there are usually rich unlabeled user behaviors that encode rich information of user interests and characteristics (Xie et al., 2020). For example, as shown in Fig. 1(a), the news click behaviors can reveal the user's interests in sports and politics. Thus, mining unlabeled user behavior data has the potential to generate a universal understanding of users to empower user modeling in downstream tasks (Wu et al., 2020a). Motivated by the great success of pre-trained language models in NLP which are first pre-trained on large-scale unlabeled corpus via self-supervision and then finetuned in downstream NLP tasks, pretraining user models on unlabeled user behavior data may improve user modeling for many userintensive tasks. In many language model pretraining methods like BERT (Devlin et al., 2019), a masked token prediction task is used to capture the contexts of words. In a similar way, predicting the masked user behavior in a user behavior sequence may help model the relations between user behaviors. For example, as shown in Fig. 1(a), by predicting the masked user behaviors, the model can capture the relations between user behaviors and further help infer the user's interest (e.g., the Warriors team). In addition, motivated by the sentence pair prediction task in language model pre-training for enhancing sentence modeling (Vaswani et al., 2017;Chi et al., 2021), matching the user behavior sequences may also be helpful for capturing user interests. As shown in Fig. 1(b), by matching the two behavior sequences from the same user in different periods, the model can better capture the inherent user interests (e.g., NFL) that is relatively stable over time.
In this paper, we propose a contrastive user model pre-training method named UserBERT. 1 In our approach, we pre-train the user models in two contrastive self-supervision tasks. The first one is masked behavior prediction (MBP), which is used to model the relatedness between user behaviors. The second one is behavior sequence matching (BSM). It requires the model to identify whether two behavior sequences in different time periods come from the same user, which aims to capture the inherent user interests that are consistent over time. Negative sampling is important for contrastive learning (Kalantidis et al., 2020). However, random negative samples may not be informative while globally hardest negatives (Xiong et al., 2020) may be too confusing. Thus, we propose a medium-hard negative sampling framework to select locally hardest samples for contrastive learning. We first randomly construct a candidate behavior pool and a behavior sequence pool from the full candidate sets, and then select the top candidates that are most similar but not identical to the targets as negative samples. We synchronously refresh the candidate behavior pool in each iteration, but asynchronously update the candidate behavior sequence pool after every certain number of iterations to reduce the computational cost. Extensive experiments on two real-world datasets validate that our approach can consistently improve various user models and outperform several state-of-the-art user model pre-training methods.
The main contributions of this paper include: • We propose a contrastive user model pre- 1 We will soon release source codes and pre-trained models.
training method that can exploit unlabeled user behaviors to pre-train user models.
• We propose a medium-hard negative sampling framework for more effective contrastive user model pre-training.
• We conduct extensive experiments on two real-world datasets to verify the effectiveness and advantage of our method.
2 Related Work

User Model Pre-training
In recent years, there are a few works on pretraining user models using unlabeled user behavior data via self-supervision (Wu et al., 2020a;Yuan et al., 2020;Xie et al., 2020). For example, Wu et al. (2020a) proposed a PTUM approach that uses a masked behavior prediction task and a next K behaviors prediction tasks to pre-train user models. Yuan et al. (2020) proposed a PeterRec approach that pre-trains user models in a masked behavior prediction task and an auto-regressive task that successively predicts user behaviors based on past ones. However, user behaviors usually have much randomness, and only predicting specific user behaviors may not be optimal for modeling the inherent user characteristics. Xie et al. (2020) proposed a contrastive user model pre-training method that augments the user behavior sequences by the crop, mask and reorder operations, and matches the augmented behavior sequences from the same user. However, in this method the augmented behavior sequences from the same users are usually highly overlapped, which may not be beneficial for the model to capture the inherent user interests. In addition, the negative samples in all these methods are randomly drawn from the entire candidate sets, which usually have huge differences with the target positive samples. These negative samples can be too easy for the model to distinguish, which may not be informative for user model pre-training. Different from these methods, our UserBERT approach incorporates a masked behavior prediction task to capture the relatedness between user behaviors and a behavior sequence matching task to capture the inherent user interests that are consistent in different periods. In addition, our approach employs a medium-hard negative sampling framework to select locally hardest negative samples, which can help learn more discriminative user models.

Negative Sampling for Contrastive Learning
Negative sampling techniques for contrastive learning have been extensively studied (Chen et al., 2020;Robinson et al., 2020). For example, Oord et al. (2018) used randomly selected negative samples for contrastive learning. Xiao et al. (2017) proposed to use a circular queue to store the features of these negative candidates that are included in recent batches. He et al. (2020) proposed to maintain a negative candidate pool of recent batches and update their representations with gradient momentum. In these methods negative candidates are in fact uniformly selected from the full candidate set, which usually have significant differences with the positive samples and may be too easy for the model to discriminate. Researchers found that many of these negatives are not informative for model training (Kalantidis et al., 2020). Thus, several methods explore to select hard negatives for training discriminative models (Xiong et al., 2020;Robinson et al., 2020). For example, Xiong et al. (2020) proposed to use an asynchronously updated ANN index to select the globally hardest negative samples. However, globally hardest samples may be too confusing for the model and can sometimes be misleading, which may be suboptimal. In our approach, we propose a medium-hard negative sampling method to select locally hardest negatives, which may be more appropriate for learning an accurate and discriminative model.

General User Model Framework
We first briefly introduce the general framework of user model, as shown in Fig. 2. It leverages a hierarchical architecture with a behavior encoder and a user encoder. The behavior encoder transforms each user behavior and its position in the behavior sequence into its embedding. It can be implemented by various models according to the characteristics of user behaviors. For example, for ID-based user behaviors, it can be implemented with an ID embedding table (Sun et al., 2019). For user behaviors that involve textual information, it can be neural NLP models like CNN (Kim, 2014) and Transformer (Vaswani et al., 2017) networks. The user encoder takes the behavior embedding sequence as input and encodes it into a user embedding. It contains two submodules, i.e., a behavior context encoder to learn hidden behavior  representations by capturing the contexts of behaviors, and a behavior aggregator that summarizes the contextual behavior representations into a unified user embedding. The behavior context encoder can be implemented by many models like CNN (Wu et al., 2019d), LSTM  and self-attention (Wu et al., 2019c) networks, and the behavior aggregator can be a max pooling, average pooling, attentive pooling (Yang et al., 2016) or last pooling module (Hidasi et al., 2016). By pre-training the user model with unlabeled user behaviors via self-supervision, the model can exploit the universal user information conveyed by user behaviors to empower downstream tasks.

Contrastive Pre-training Tasks
Next, we introduce the two self-supervision tasks for contrastive user model pre-training, i.e., masked behavior prediction (MBP) and behavior sequence matching (BSM).

Masked Behavior Prediction (MBP)
Following PTUM (Wu et al., 2020a), we use a masked behavior prediction task with small variants to capture the relations between behaviors of the same user, as shown in Fig. 3(a). In PTUM the masked behavior is predicted from the user embedding that does not encode masked positions. Instead, we use the hidden representations (generated by the behavior context encoder) at the masked positions to predict the masked behavior to better capture masked behaviors' positions. We randomly mask 10% of user behaviors (and at least one) in a user behavior sequence, and pre-train the model in a contrastive way. For each masked behavior x (regarded as the positive candidate) we sample K negative candidate behaviors, and we use a matcher to jointly predict the matching scores of these K + 1 candidates based on their relevance to  the hidden representation at the masking position. These scores are further normalized by softmax, and the task is formulated as a K + 1-way classification problem. The loss function we used is cross-entropy, which is formulated as follows: where y x i andŷ x i are gold and predicted labels of the i-th candidate behavior given the masked behavior x, and M is the set of masked behaviors.

Behavior Sequence Matching (BSM)
The second pre-training task is behavior sequence matching (BSM). As shown in Fig. 3(b), the goal of this task is to identify whether two behavior sequences A and B come from the same user. To enforce the model to capture the inherent user interests that are relatively stable across different time periods, we ensure that the behavior sequences A and B have no overlap in time. We use the user model to encode both sequences and evaluate their similarity via a matcher. For each pair of behavior sequences from the same user (sequence B is regarded as the positive candidate), we sample P negative candidate behavior sequences from other users that do not overlap with sequence A in time. We jointly predict the matching scores of the P + 1 behavior sequences and normalize them via softmax. The loss function of the behavior sequence matching task is also cross-entropy, which is formulated as follows: where y A i andŷ A i are the gold label and predicted score of the i-th candidate behavior sequence given the target sequence A, and S is the set of user behavior sequences for model pre-training. We jointly pre-train the user model in both MBP and BSM tasks, and the unified loss function L is a summation of the two loss functions as follows:

Medium-Hard Negative Sampling
Finally, we introduce our proposed medium-hard negative sampling method to select the locally hardest negative candidate behaviors and candidate behavior sequences for contrastive pre-training. As shown in Fig. 4, it contains two modules, i.e., negative behavior sampling (right) and negative user behavior sequence sampling (left). The negative behavior sampling module aims to select negative candidates for the masked behavior prediction task. Negatives uniformly selected from the full behavior set may not be informative because they are usually easy to be distinguished. However, the globally hardest negatives may be too puzzling or even misleading to the model. For example, the hardest negative for a target webpage browsing behavior of "NFL Schedule: Week 1" can be a webpage entitled "NFL Schedule | Week 1", which has no substantial difference with the target. To handle this problem, we propose to construct a smaller candidate behavior pool and select the locally hardest samples from this pool as "medium-hard" negatives. We assume that the entire behavior set B contains M user behaviors, and we randomly sample m (m << M ) behaviors to construct a candidate behavior pool. This pool is synchronously updated during the model training, i.e., re-sampling user behaviors from B at each model training step. We denote a masked behavior at the i-th model  We use the behavior encoder in the user model to encode these behaviors into their hidden representations, which are respectively denoted as h We evaluate the cosine similarities between h x i and each behavior embedding in H b , and select K behaviors with top similarity scores as negative candidates for masked behavior prediction (candidates that are identical to the target are filtered). In this way, the negatives are harder if m is larger. We can adjust the difficulties of negative samples by tuning the value of m.
The negative user behavior sequence sampling module aims to select negative candidates for the behavior sequence matching task, which uses a similar way to sample medium-hard negatives by choosing the locally hardest samples in a candidate behavior sequence pool. However, different from processing user behaviors, encoding behavior sequences is much more time-consuming. Thus, we construct a behavior sequence pool that is asynchronously updated after every certain number of iterations to reduce the computational cost. We denote the target user behavior sequence from the i-th to the j-th training steps as [A i , ..., A j ]. We randomly crop u behavior sequences from the behavior sequences of other users in the full user set to form the candidate behavior sequence pool, which is denoted as [B 1 , ..., B u ]. We ensure that these sequences do not have overlaps with [A i , ..., A j ] in time. This pool is static between the i-th and j-th training steps (the interval is j−i+1 steps). We use the user model at the i-th training step to encode target and candidate behavior sequences, which are respectively denoted as R . We use cosine distance to measure the similarity between each target behavior sequence embedding in R A and each candidate in R B , and select the candidates with the highest similarities to the target sequence as negative samples. 2 In this way, we can obtain medium-hard candidate user behavior sequences efficiently by using a moderate pool size and update interval.

Datasets and Experimental Settings
In our experiments, we conduct experiments in two tasks. The first task is news recommendation. We take the dataset (denoted as News) used in . It contains news click behaviors of 10,000 users on Microsoft News and webpages browsing behaviors on Bing in one month. As studied in , the goal is to predict future news clicks of users based on the titles of their browsed webpages. The second task is Ad CTR prediction. The dataset (denoted as CTR) contains Ad titles and descriptions, Ad impressions, and the webpage browsing behaviors of 374,584 users in one month, which is used by (Wu et al., 2020a).   The goal is to predict whether a user clicks a candidate Ad based on Ad texts and the titles of this user's browsed webpages. In the two tasks, we used the same dataset split strategy as prior works (Wu et al., , 2020a We used the same unlabeled user behavior dataset for user model pre-training, which contains the titles of browsed webpages of 500,000 users on Bing in six months. The dataset statistics are summarized in Table 1. In our experiments, for fair comparison we followed the base user models settings in PTUM (Wu et al., 2020a). We used dot product to implement the matchers in our approach. The number of negative candidates was 4. The sizes of candidate behavior and behavior sequence pools were 1,000 and 100, respectively. The behavior sequence pool update interval was 50 steps. We used Adam (Kingma and Ba, 2015) as the optimizer. The learning rate was 1e-4. Detailed hyperparameter configurations are included in supplementary materials. These hyperparameters were tuned according to the validation performance. We used AUC and nDCG@10 to measure model performance on the News dataset, and used AUC and AP as metrics on the CTR dataset. We repeated each experiment 5 times and reported the average results.

Performance Evaluation
In this section, we compare the performance of our approach with several baseline methods, including (a) w/o pre-training; (b) PTUM (Wu et al., 2020a) and (c) the contrastive pre-training method (denoted as CP) proposed in (Xie et al., 2020). On the News dataset we use NAML , LSTUR , and NRMS (Wu et al., 2019c), which are widely used news recommendation methods (Wu et al., 2020b). on the CTR dataset we use GRU4Rec (Hidasi et al., 2016), Na-tiveCTR  and BERT4Rec (Sun et al., 2019) to implement the user models. The performance under different percentage of training data on the two datasets is respectively shown in Tables 2 and 3, which leads to several findings. First, pre-trained user models consistently achieve better performance than the models without pre-training, and the advantage is larger when less labeled data is used for training. This is because pre-trained user models can exploit universal user information encoded by unlabeled user behaviors to enhance user modeling, which can reduce the dependency of user models on labeled data. Second, PTUM and UserBERT achieve better performance than CP. This may be because in the CP method, the behavior sequences augmented from the same user may have many overlaps, which is not beneficial for the model to capture the inherent user interests. Third, UserBERT consistently outperforms PTUM and the improvement is significant (p < 0.05 in t-tests). This is probably because PTUM uses selfsupervision tasks that only predict specific behaviors, which may be disturbed by the randomness of user behaviors. Our UserBERT approach uses a behavior sequence matching task to capture the intrinsic user interests, which may be more robust to the noisy behaviors. In addition, in PTUM negative samples are randomly selected, which may not be very informative nor representative. In contrast, UserBERT is trained with more informative negatives with a medium-hard negative sampling method, which yields better performance.

Influence of Pre-training Tasks
In this section, we study the influence of the selfsupervision tasks for user model pre-training. We first verify the effectiveness of each task used in UserBERT. We compare the results of NRMS on the News dataset and BERT4Rec on the CTR dataset without pre-training or pre-trained with different tasks. 3 Due to space limitation, we only present the results on the News dataset in the main content, and the results on the CTR dataset are in supplements. From the results shown in Fig. 5, we find that both BSM and MBP tasks are very useful for model pre-training. This may be because the MBP task can encourage the user model to capture the relatedness between behaviors and the BSM task helps to model the inherent user interests that are consistent across different time periods. In addition, combining both tasks is better than using a single one. It shows that the two tasks can provide complementary information for each other to improve user model pre-training.
Then, we compare our newly proposed BSM task with the next K behaviors prediction (NBP) task from PTUM. We evaluate the model performance using one of the NBP and BSM tasks or combining both of them (the MBP task is still preserved). The results are shown in Fig. 6. We find the models pre-trained in the BSM task outperforms those pre-trained in the NBP task. This may be because the NBP task in fact has some overlaps with the MBP task (can be regarded as masking the last K behaviors), and predicting the exact behaviors will also be disturbed by the behavior randomness. In contrast, the BSM task can also help the model capture the relatedness between past and future behaviors and may be more robust to the noisy behaviors. In addition, we do not observe any performance gain when combining the BSM and NBP tasks. Thus, we only use the MBP and BSM tasks in our approach.

Effectiveness of Medium-Hard Negatives
In this section, we verify the effectiveness of our proposed medium-hard negative sampling framework. We compare the model performance using random, globally hardest and medium-hard negative candidates, as shown in Fig. 7. From the results, we find that using globally hardest negatives is slightly better than using random negatives on the News dataset. However, the performance is worse than using random ones on the CTR dataset (see supplements). This may be because the globally hardest negatives may be too difficult for the model to distinguish and may even be misleading. Our proposed medium-hard negative sampling method outperforms random and hardest negative sampling. It shows that medium-hard negative candidates are more suitable for user model pre-training.

Hyperparameter Analysis
In this section, we present some analysis on several key hyperparameters in our approach. We first study the influence of the size of candidate behavior and behavior sequence pools on the model performance, and the results are respectively shown in Figs. 8 and 9. We find that when the pool sizes are too small, the performance is not optimal because the hardest negative samples in a small candidate pool may still be easy samples. However, when the pool sizes get too large, the performance also declines. This is because the selected samples will get close to the globally hardest samples, which are  too difficult to be distinguished. Thus, we choose moderate values for the two pool sizes, i.e., 1,000 for the candidate behavior pool and 100 for the candidate behavior sequence pool. Then, we study the influence of the update interval of the candidate behavior sequence pool. The model performance, as well as the sampling and overall pre-training time per step under different update intervals are shown in Fig. 10. We find that a smaller update interval usually yields slightly better performance, but the sampling computational cost is also larger. The improvement is quite marginal when the interval is smaller than 50 steps while the computational cost grows rapidly. Thus, we set the update interval to 50 steps to achieve a good trade-off between performance and efficiency.

Conclusion
In this paper, we propose a UserBERT approach that contrastively pre-trains user models in two selfsupervision tasks, i.e., a masked behavior prediction task to capture relations between user behaviors and a behavior sequence matching task to capture the inherent user interests. In addition, we propose a medium-hard negative sampling framework to select informative negative candidates to learn accurate and discriminative user models. Extensive experiments on two real-world datasets show that UserBERT can consistently improve many user models and outperform several baseline methods.   Figure 16: Influence of the update interval of behavior sequence pool on the model performance.