{'fraction_non_alphanumeric': 0.035307849430989205, 'fraction_numerical': 0.007440910417274584, 'mean_word_length': 4.957844415471534, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
MV-HAN: A Hybrid Attentive Networks based Multi-View Learning Model for Large-scale Contents Recommendation

Industrial recommender systems usually employ multi-source data to improve the recommendation quality, while effectively sharing information between different data sources remain a challenge. In this paper, we introduce a novel Multi-View Approach with Hybrid Attentive Networks (MV-HAN) for contents retrieval at the matching stage of recommender systems. The proposed model enables high-order feature interaction from various input features while effectively transferring knowledge between different types. By employing a well-placed parameters sharing strategy, the MV-HAN substantially improves the retrieval performance in sparse types. The designed MV-HAN inherits the efficiency advantages in the online service from the two-tower model, by mapping users and contents of different types into the same features space. This enables fast retrieval of similar contents with an approximate nearest neighbor algorithm. We conduct offline experiments on several industrial datasets, demonstrating that the proposed MV-HAN significantly outperforms baselines on the content retrieval tasks. Importantly, the MV-HAN is deployed in a real-world matching system. Online A/B test results show that the proposed method can significantly improve the quality of recommendations.


INTRODUCTION
Personalized recommender systems have been widely employed in web applications, such as e-commerce services, news recommendations, and video recommendations [1,2,5,16]. The recommender systems improve user experience by filtering items in which users are interested. In the industrial scenario, scoring large-scale items effectively in real-time becomes challenging, as the system serves billion-scale users with billion-scale contents, e.g., QQ Kandian. There is a general practice that designs the whole system with a matching stage and a ranking stage, as shown in Figure 1. The matching stage aims at retrieving hundreds or thousands of satisfying items from billions of candidates. Next, the ranking stage generates a meticulous ranking list based on the selected items. Both matching and ranking stages play critical roles in the entire recommendation pipeline. In this work, we target on the matching stage.
Inspired by recent success of deep learning achieved in other domains [7,14,17,21,22], many research applies deep neural networks (DNNs) to personalized recommender systems [1,2,4]. In particular, the two-tower model (TTM) is one of the most popular DNN-based methods deployed in the real-world matching stage [8,9,20]. TTM learns two mappings for users and item features via two independent DNN towers, encoding representations of users and items in the same space. This is beneficial to the online serving module, as similar items can be efficiently retrieved with an approximate nearest neighbor (ANN) algorithm [10]. However, though the TTM-based models are effective, there remain several challenges to be addressed. Namely, Though there exists some research attacking these issues (e.g., [6,12,15,18]), only few of them address all problems simultaneously, and are deployed in real-world large-scale recommender systems.
To tackle the above challenges, we propose Multi-View Hybrid Attentive Networks (MV-HAN), for the matching stage in industrial recommender systems. The MV-HAN extends the TTM by transferring information between different content types via several hybrid neural networks. The proposed method shares parameters of the bottom structures between different data types, which enables information sharing effectively. As such, the minor data types and features are both well-trained. Our MV-HAN is implemented with several multi-head self-attentive neural networks with residual connections, which promotes feature interactions. This enables knowledge to be transferred efficiently among different content types, and allows to automatically process multiple types of data from asynchronous pipelines.
To summarize, this paper makes the following contributions: • We proposed a generic modeling framework for the matching stage in the recommender systems by extending the two-tower model for transferring source information to the target type.

PROPOSED METHOD 2.1 Problem Statement
In this work, we focus on the matching stage in recommender systems. The issue in the matching stage is a typical Information Retrieval (IR) question, aiming at retrieving a set of content that users are interested in from massive content. We consider a user and a target content set O = [ 1 , 2 , ...., ], where , and denote the number of users, source contents and target contents respectively. The user-content interaction history can be defined as a matrix Y ∈ R ×( + ) , where = 1 if the interaction between user and content is observed, and = 0 otherwise. Our object is to learn a functionˆ= ( , |Θ) to predict the scoreˆof interaction , where Θ denotes model parameters. In this way, we can rank relevant content by prediction scores.

Multi-View Hybrid Neural Networks
We show the overall architecture of the MV-HAN in Figure 2. The MV-HAN contains two model towers for users and contents respectively. This inherits from TTM to keep the embedding of users and contents independently and retrieves similar contents in the online serving stage efficiently. Each tower of the MV-HAN includes three major structures: embedding layers, feature extraction layers, and multi-view representation layers. The users' tower can learn better representations by sharing parameters even if the data of the target types is limited. Different from users' towers, the content tower merely shares parameters in embedding layers and feature extracting layers. This is to balance the learning process between different types of content to mitigate the cold start issue in the sparse types. In addition, feature extraction layers are implemented with multi-head self-attentive neural (MHSA) networks with residual connections, which extract high-order latent feature interactions effectively [15,17]. The multi-view representation layers are several independent MLPs, capturing the difference between source and target contents. Finally, the MV-HAN predicts the final  Figure 2: An overview of the proposed MV-HAN.
scores by the users' and contents' representations. We formulate overall MV-HAN as follows: Here denotes the -th user's representation, , and denote the -th, and -th content's representations of source and target data. , , and denote the mappings of multi-view representation layers with users, source contents, and target contents, which are implemented by several MLPs. and denote the mappings of feature extraction layers, which are implemented by multiple blocks via MHSA networks with residual connections. The , , and are formulated as: where includes the embedding and concatenate function for users, and for contents. We follow a common setting to predict the user-content interaction scores by the cosine function. Specifically: whereˆandˆdenote the prediction score of the source type and the target type.

Optimization Objective
The retrieval problem is essentially a classification problem, which estimates the probability distribution ofˆwith a softmax function: .
The loss for user can be formulated as: Since is usually huge in industrial scenarios, it is time-consuming to include all contents in computing Eq. 5. To address this problem, for each user-content interaction, we randomly sample negative cases in = 0 for user . Then we use Stochastic Gradient Decent (SGD) and its variants to optimize multiple objectives. We employ an alternative way to train the proposed method. Specifically, we put pairs with the same type in a batch, e.g., the source pair ( , , ), and update model parameters with this batch. Next, we train the model with data of another type to update model parameters.

EXPERIMENT
We conduct both online and offline experiments to evaluate the performance of the proposed MV-HAN, to answer the following research questions: To answer RQ1, we conduct offline experiments on several industrial datasets to compare the MV-HAN with baselines.
To answer RQ2, we conduct ablation studies to evaluate our purpose-built designs.
To answer RQ3, we deploy MV-HAN in our real-world application and conduct an online A/B test to evaluate the performance of the proposed model.  • Articles is a mature type in our platform, including texts and images. We choose it as the source dataset since it contains rich information about users. • Novels is a new type in QQ Kandian. We collect user behaviors from this type as the target dataset to evaluate the performance of methods. • Short Contents (SC) is a relatively sparse type since short contents usually are released and taken down fast. The data of short contents are selected as a different target source.

Baselines Methods.
We compare the proposed method with several existing recommendation methods used in the matching stage, including logistics regression (LR) [13], YoutubeDNN [2], Two-Tower based Model (TTM) [20], Cross-domain Content-boosted Collaborative Filtering neural NETwork (CCCFNet) [11] and Multi-View Deep Neural Network (MV-DNN) [3]. We train TTM on both source and target datasets to evaluate the performance of knowledge transfer. We mark this method as TTM_all.

Metrics.
We employ two widely used metrics, Area Under the ROC Curve (AUC) and Hit Ratio (HR@50) [12,19] to evaluate the performance of the MV-HAN. The AUC is used for evaluating the ranking performance of contents, and the HR is used for testing whether good contents are retrieved. Note that we use the relative AUC improvement [5,23] for the evaluation in this paper.  Table 2. Observe that our proposed MV-HAN consistently outperforms all baseline methods on both datasets. Specifically, MV-HAN achieves up to 93.12% higher relative AUC and 113.45% higher relative HR. This verifies the strong ability of interested content retrieval for our model. In addition, though TTM_all and MV-DNN can utilize the source information to mitigate the cold start problem, the vanilla TTM outperforms those methods in the Novel dataset. This demonstrates that simply utilizing source data may dilute the knowledge of the target type. MV-HAN also achieves the best results in this dataset, which proves that the proposed method is more effective for transferring the latent information between different data types.  Table 3. Observe that the MV-HAN achieves the best performance in both datasets, which verifies the effectiveness of the purpose-built design.
Specifically, compared to the MV-HAN w/o SE, the MV-HAN obtains substantial relative improvements in SC over the Novel dataset. This is because the short contents are updated more agilely than in novels. Thus, the model becomes underfitting over this data type. In contrast, compared to the MV-HAN w/o FE, MV-HAN achieves greater relative improvements in Novel than in the SC dataset. As the novel type includes the smallest number of training data, the corresponding tower may be trained insufficiently. In addition, compared to MV-HAN , the MV-HAN also achieves 1.07% ∼ 2.08% relative improvements in the two datasets. This demonstrates that the multi-head self-attentive neural networks have better feature extraction ability over MLPs.

Online A/B Test
In order to evaluate the performance of the proposed method in the real-world application, we deploy MV-HAN in the QQ Kandian short content recommendation and conducted live experiments to compare MV-HAN with a two-tower based method only trained by the short content data. The online evaluation metrics employed for evaluation are Click-Through-Rate (CTR), the number of the Daily Active User (DAU), the number of the clicks and the Duration users used. Table 4 shows the relative changes after employing our method. Observe that the proposed MV-HAN obtains significant improvements in all metrics. Specifically, the MV-HAN achieves 6.95% and 10.12% relative improvement in terms of CTR and #Clicks, respectively. This is because MV-HAN can transfer information from other types to the short content to retrieve contents in which users are more interested. DAU and Duration are regarded as key metrics for respectively evaluating the short-term and long-term competitiveness of a platform, especially for user-generated content (UGC) applications. The MV-HAN gains relative improvement of 5.09% and 9.69% in DAU and Duration respectively. It shows that the MV-HAN is beneficial in keeping users for both the short term and the long term.

CONCLUSION
In this paper, we propose a novel model called MV-HAN for the matching stage in recommender systems. We design a hybrid neural structure configured with different models, including MLPs and multi-head self-attentive neural networks. The proposed method transfers the knowledge from the source types to the target types, which helps better representation learning for users and contents. Moreover, the MV-HAN shares parameters of the bottom networks to mitigate the cold start on the spare types. Offline experiment results on industrial datasets show that the proposed method outperforms different baselines, i.e., achieving up to 4.43% and 4.64% higher AUC and HR than the best results of baseline methods on the SC dataset. Online experiment results on real-world recommender systems show that the MV-HAN significantly improves the recommendation performance compared with baseline methods in all metrics. It verifies that the MV-HAN is able to handle multi-source asynchronous dataflows and extract information from different content types in real-world applications.