{'fraction_non_alphanumeric': 0.02746815286624204, 'fraction_numerical': 0.01101380042462845, 'mean_word_length': 5.027459344174886, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
PHN: Parallel heterogeneous network with soft gating for CTR prediction

The Click-though Rate (CTR) prediction task is a basic task in recommendation system. Most of the previous researches of CTR models built based on Wide \&deep structure and gradually evolved into parallel structures with different modules. However, the simple accumulation of parallel structures can lead to higher structural complexity and longer training time. Based on the Sigmoid activation function of output layer, the linear addition activation value of parallel structures in the training process is easy to make the samples fall into the weak gradient interval, resulting in the phenomenon of weak gradient, and reducing the effectiveness of training. To this end, this paper proposes a Parallel Heterogeneous Network (PHN) model, which constructs a network with parallel structure through three different interaction analysis methods, and uses Soft Selection Gating (SSG) to feature heterogeneous data with different structure. Finally, residual link with trainable parameters are used in the network to mitigate the influence of weak gradient phenomenon. Furthermore, we demonstrate the effectiveness of PHN in a large number of comparative experiments, and visualize the performance of the model in training process and structure.


INTRODUCTION
Recommendation system has provided to people a high quality services and brings objective economic benefits to the company. The Click-through rate (CTR) prediction is one of the important basic tasks in recommendation system. By predict the click rate of user, the web or application can sort the candidate item list and push them to target user, so as to provide personalized recommendation service for users. Early CTR prediction models, usually in a simple form, output CTR through Logistic Regression, and use automatic feature engineering methods such as Factorization Machine(FM) [Rendle 2010] and Gradient Boosting Decision Tree(GBDT) [Friedman 2001] for business implementation. With the development of deep learning, CTR prediction model based on neural network has gradually become the mainstream application model in the real application. The CTR prediction model based deep learning can roughly classify into two categories: one is Wide & Deep[Cheng et al. 2016] structure, which is analyzed separately based on fixed features. The other is a DIN ] structure based on the user's historical behavior. DIN is a model for analyzing user's historical behavior, which predicts CTR by calculating the amount of attention between the target item and the user's history item. In the later development of such models, combined with different analysis structures, including DIEN [Zhou et al. 2019], BST [Chen et al. 2019] and DIHN [Shen et al. 2022] models. However, they generally relied on the historical behavior of users and needed some other strategies or structures to supplement the problem of cold start.Wide&deep structure have used parallel structures of different depths to consider both memorization and generalization. In subsequent studies, FNN , DeepCrossing [Shan et al. 2016], DeepFM [Guo et al. 2017], DCN [Wang et al. 2017], xDeepFM [Lian et al. 2018], DCN-V2 [Wang et al. 2021], EDCN [Chen et al. 2021] and other models have similar parallel structure like Wide&deep, and were utilized to analyze public embedding through different modules. Moreover, the generalization of this structure depended on the effectiveness of parallel structures.
At the end of CTR model, the click rate prediction output have been achieved by linear layer with activation function. During training phase, the activation values between parallel layers tend to arXiv:2206.09184v1 [cs.
LG] 18 Jun 2022 fall into the weak gradient interval. This phenomenon will weaken the training effect of each parallel module, and can not improve the generalization while improving the complexity of the model.
In this paper, we propose a new deep CTR model, named Parallel Heterogeneous Network (PHN). For PHN model, three parallel feature interaction structures were included to analyze CTR features from different perspectives. In order to enhance the independent analysis ability of each parallel module, Soft Select Gating module was constructed after public embedding to enhance the original embedding expression. We also added residual connections with trainable parameters to the model to reduce the weak gradient phenomenon by accumulating gradients during the back propagation process.
This paper mainly contributions are as follow: • In order to strengthen the expression ability of CTR prediction model, A constructed three different linear feature interaction methods from nonlinear interaction, bite-wise interaction and vector-wise interaction based on parallel structure. • Soft Selection Gating is constructed before the parallel structure, and the features of original embedding are enhanced by self-attention and soft gate structure while retaining the high order crossover characteristics, which improves the ability of the model to express data. • To solve the weak gradient phenomenon in the parallel model, the residual link with trainable parameters are used in the parallel structure to reinforce the model training process. • The effectiveness of Soft Selection Gating and the weak gradient phenomenon are visualized, and the effectiveness of PHN is verified by comparison experiments.
The rest of the contents of this paper are as follow: Part 2 introduce the relevant work of this paper, including the information selection and the feature interaction in CTR model study. Part 3 focus on the overall structure and details of the PHN model. Part 4 introduce the experimental purpose and relevant preparation, and verify the effectiveness of the model through the experimental results. Part 5 is the conclusion.

RELATED WORK
In CTR prediction model study, there are two important parts.
Information selection: In the early development of CTR model, feature engineering, as the main means to improve business effect, was focused on personalized modeling user interest. And the simple model based on logistic regression was utilized to perform the leaf node output of GBDT[Friedman 2001] as the second-order feature to effectively improve the fitting effect of the model. In a similar fashion, the linear model based on second-order interaction also used FM[Rendle 2010], FwFM [Pan et al. 2018], and FmFM [Sun et al. 2021] to map sparse second-order features to enhance the training effect of the model. Morerecently, CTR has also payed more attention to feature engineering. The feature engineering models based on FM include FNN  and DeepFM [Guo et al. 2017]. Moreover, the method of using feature point multiplication in PNN [Qu et al. 2016] was also realized. The SENET [Hu et al. 2018] part of FiBiNet ] was regarded as feature processing. The L0-SIGN [Su et al. 2021] uses graph neural network to select information and feature intersection.
Feature Interaction: Based on the background knowledge of the recommendation system, the combination of features has been used effectively to analyze the preferences of different user groups for specific products, which makes the feature interaction as a key part of CTR prediction model. In a more advanced method such as PNN [Qu et al. 2016], DCN [Wang et al. 2017] and DCNV2 [Wang et al. 2021], NFM [He and Chua 2017], FiBiNet  and FINT [Zhao et al. 2021], xDeepFM [Lian et al. 2018], and Au-toInt [Song et al. 2019] were separated using for interaction patterns to product layer, cross layer, Hadamard product, field interation, compressed interation network, and Multi-head Self-Attention [Vaswani et al. 2017], respectively.

PROPOSED METHOD
The Parallel Heterogeneous Network (PHN) consists of two main structures. One of them is Soft Selection Gating (SSG) module based on self-attention to enhanced embedding features for different structures, and another one is Heterogeneous Interaction Layer(HIL), using different interaction method to analyze the enhanced features, and finally using Logistic Regression to output the confidence of sample. Fig.1 illustrates the main structure and the detail of model.

Soft-Gating Information Selection
The parallel structure models aim to achieve better generalization by specifying different features interaction structures in a parallel way. This kind of model is similar to bagging model, which means that we can apply more than two structure in CTR model.
The ideal of multiple structure parallelism raise up a new question: whether different structures require different input dense feature. The traditional Multi-head self-attention(MSA) mechanism [Vaswani et al. 2017] used weight based query vector and key vector to aggregate information in a sequence, which was an ideal method to process feature information.
From the Eq12, MSA has considered different field weights through the second-order intersection of query vector and key vector . However, based on the feature interaction in the CTR prediction model, the direct using by the traditional MSA may over-focus on the feature activation value of the second-order crossover, thus losing the performance of the feature at the higher-order crossover. Inspired by the FRNet [Wang et al. 2022], an information selecting method named Soft Selection Gating (SSG) is used after the sharing embedding ∈ in PHN. This soft-gating information selection is designed for choosing activation between MSA result and sharing raw embedding.
where ∈ is the trainable gating vector, ∈ is the sharing self-attention embedding, ∈ is the sharing embedding, and ∈ is a unit vector. The SSG considers both sharing embedding and self-attention embedding. By using weighting parameters, the model can select the raw feature or the feature enhanced by MSA for different parallel structures. Subsequent experiments will discuss whether to share the weight of self-attention and the gating mechanism, and confirm the effectiveness of SSG.

Heterogeneous Interaction Layer
In PHN, the parallel use three kinds of interaction layers to implementation the parallel structure: 1) cross layer is the basic part of DCN [Wang et al. 2017] and DCNV2 [Wang et al. 2021], which focuses on the bit-aware feature interaction; 2) field interaction layer is the basic part of FINT [Zhao et al. 2021], which focuses on the vector-aware field interaction; 3) feed forward layer is used to fitting the non-polynomial information.
3.2.1 Cross Interaction Layer. Feature interaction is a main key point in study of mainly CTR prediction model. As a previous study, the DCN[Wang et al. 2017] and the DCNV2 [Wang et al. 2021] proposed two kinds of explicit interaction methods, which achieved the data mode of high-order interaction by realizing the intersection of multi-layer hidden features and original features.
where, 0 is the input feature of the first cross layer; is the output feature of the i-th cross layer, and represent trainable parameter vectors and matrices; ⊙ means Hadamard product and × means matrix multiplication. In Eq.?? the DCN and DCNV2 used different parameter forms to interact features, but in general, it achieves bit-aware feature interaction. The PHN combines the formulas of DCN and DCNV2 to construct the bit-aware interaction module.

Figure 2: The calculation diagram of cross layers in PHN
As mentioned in Fig.2, we use the parameter part of the two crossover model and bias to construct cross layer in PHN.

Field Interaction Layer.
Besides bit-aware interaction, vectoraware interaction is also a key part of the model construction. Field Interaction module is mentioned in FiBiNet ] and FINT [Zhao et al. 2021], which using the cross method to implement the vector-aware interaction. PHN use the Field Interaction layer in FRNet as a parallel part to enhance the generalization effect of the whole network on the feature crossing pattern.
As shown in Fig.3, field interaction layer uses the residual link with trainable parameter vector, which product on different fields feature to screen the output features of the upper layer. In the where, is the activation function, which is LeakyReLU in PHN.

Weak Gradient Problem
Basic on the prediction task definition, the key point of improving AUC value is increasing the confidence of positive label samples and decreasing the confidence of negative label samples, which makes model more robust. In the last stage of CTR prediction model, the traditional model usually constructs confidence coefficient of click by using Sigmoid activation function.
As shown in Eq.??, as the absolute value of the final linear layer output increases, the gradient propagated back based on the activation function also decreases.

Figure 4: Gradients of different interval in Sigmoid
The Fig.4 shows that, we can think of the entire Sigmoid function as two interval, the effective gradient interval (bule) with a normal gradient, and the Weak gradient interval (red) with a gradient approaching zero. When the output value of the parallel model is accumulated at the last linear layer, it is easy to make the samples originally in the effective gradient interval to fall into the Weak gradient interval, thus weakening the learning of each part for valid samples. In this paper, this phenomenon is referred to weak gradient in parallel structures.
To mitigate this phenomenon and achieve effective training, the PHN using residual link in each substructure, enhancing the gradient accumulation of each parallel structure in the process of back propagation. To further accommodate this phenomenon, we also tried to add gating parameters to residual links and used batch normalization of different modes in the final linear layer. We will discuss this further in Part4.

EXPERIMENTS
In this section, we evaluate PHN on two benchmark data sets. We aim to answer the following research questions: • RQ1: Will parallel structure-based PHNS perform better than previous CTR prediction models over different classical data sets? • RQ2: Is the parallel structure actually caused the weak gradient phenomenon, and the problem is effectively alleviated by residual connection or batch normalization? • RQ3: Under what circumstances can the information selecting module based on soft gating machine reasonably enhance the function of feature representation.

Datasets Description.
To evaluate the effectiveness of the model in this paper, two benchmark off-line datasets are selected for experiment: Criteo 1 and Avazu 2 . Detailed information on the two benchmark datasets is shown in Table1. For the CTR prediction task, two classical evaluation metrics such as: Logloss and AUC, were used to verify the effective generalization and robustness of the suggested model.

Experimental Setting.
In the rest of this section, there are three sub-sections to evaluate the validity of the proposed model, the weak gradient problem solution, and the Soft Selection Gating module, while the experimental model designs are different. In section4.4, the same number of cross layers were used to compare the performance of PHN and other CTR model on the benchmark data set, and grid search was performed on the number of cross layers. In section4.5, we first explored the effectiveness of RL and BN on PHN through comparative experiments. Then, different parallel parts of the PHN and different model patterns were trained an epoch independently, and 200 samples from the model were extracted for visualization of the results. In section4.6, the two benchmark models verify the validity of SSG in different modes and visually display the calculation results of SSG. All experiments were carried out on FuxiCTR [Zhu et al. 2020] open source framework.

Performance Comparison (RQ1)
4.4.1 Effectiveness of PHN.. To verify the validity of the model, we followed the original structure of each comparison model, controlled the cross layer of all models in three layers, and recorded the validation and testing results of each model on two benchmark datasets. The specific experimental results were reported in Table2. The experiment shows that, PHN achieved a good performance in both validation and testing experiments with two benchmark large datasets.

Grid
Search. Fig5 shows the grid search experiment result of PHN. With the increase of the number of cross layers, the robustness of the model also increases, which may benefit from the improvement of the expression ability of cross layers for higherorder crosses. However, as the number of layers increases, the model complexity also increase, which slow down the training process of the model. The values of AUC and Logloss shown in Fig5 tend to be stable when the number of cross layers is 5, so we can consider the optimal number of cross layers for PHN to be 5.

Weak Gradient Phenomenon (RQ2)
4.5.1 Efficiency analysis. To reduce the impact of weak gradient phenomenon, there has been an attempt in PHN to enhance the data gradient flow in training through residual links or batch normalization to reduce the training pressure of each parallel part. In experiments, we tried to introduce gating parameters for residual links and discussed the independence of batch normalization in different parallel modules. The Fig6 shows the batch normalization in different modes.
This part of the comparison test based on different solutions, gives the performance of the algorithm in different cases. Table3 shows the result of comparison experiments.
Different model subscripts represent different structures. means the basic model, means normal residual links is added to the PHN, means parameter residual links is added to the PHN, means batch normalization is followed before the final full connection layer, and means private batch normalization.  The experiments show that, In the reverse transmission process, the weak gradient problem can be improved by using gradient accumulation in the residual link. In the case of parameters, the overall network can better fit the data flow in the feed forward and reverse process, and strengthen the fitting effect of different parallel structures. However, the subsequent addition of Batch normalization has not been very effective. This may be due to the uneven distribution of data flows in different parallel structures, but forced unification with normalization weakens the representation of data. This also explains to some extent that batch normalization layer is not separable from linear layer. The last two groups of experiments also showed that the specificity of the data stream fitted was enhanced when the residual link strengthened different parallel structures, while BN had certain side effects. Therefore, in PHN, the residual and Batch Normalization had better be realized in an independent parallel structure.

Visualization of activation value.
A more robust model should output more closely to the confidence of the label worthiness. Fig.7 shows the confidence curve of PHN in 200 samples after training one epoch in different configurations.
The red line in Fig.7 represents the PHN model without residual link and batch normalization. Fig.7(a) shows that, the single crossover structure showed higher negative confidence and lower positive confidence than PHN. This shows that PHN is superior to partial cross structure in sample resolution. Fig.7(b) shows that, the sigmoid calculation after summing up the activation values of the parallel models can show more robustness than PHN. This means that the fitting effect of a single PHN model on the data set is weakened by weak gradient phenomenon. Fig.7(c) shows that, residual connection can enhance the high confidence of negative samples, but also reduce the confidence of positive samples, and

Selection Information (RQ3)
4.6.1 Data skew visualization. Based on the trained PHN structure, we visualized the tensor amplification ratio after SSG output. As shown in Figure8, the characteristics of the three cross-layers have some similarity, such as the high proportion of field 13 and the low proportion of field 39. At the same time, the feature scaling of the three parts is somewhat different, as in field 8 and field 24. 4.6.2 Selection pattern. The SSG module is designed for enhance the representation of embedding feature, which select the feature from raw embedding and self-attention embedding. Depending on the design, the selection pattern of self-attention layer and gating layer in this module can be classified as public or private. To further validate the effectiveness of SSG, we also conducted a comparison experiment under a pure parallel structure. Different subscripts represent different selection patterns: "embed" means using public embedding feature; "sa" means using public self-attention feature based on raw embedding; "sg" means using public soft gating to enhance the self-attention feature; the subscripts with a prefix "P" means that the PHN contains private layers for each parallel layer. Table 4 shows the results of comparison experiment. The experiments show that, single self-attention layer (public of private) cannot replace the embedding feature represent, but it can help to enhance the feature by using the soft selection gating, and the AUC value of the algorithm increases by 0.123% on average. From a theoretical point of view, a feature without a high activation value in the first-order feature cannot be completely transferred, because it may show a high activation value in the high-order interaction with other features.

CONCLUSION
In this paper, we described the parallel structure of the current mainstream CTR model and the weak gradient phenomenon that must be faced in the parallel structure, and introduce a parallel structure model named Parallel Heterogeneous Network (PHN) in response to these phenomena. PHN model used Soft Selecting Gating (SSG) structure to isomerize features, and used Feed Forward network, cross interaction layers and field interaction layers to build the subsequent parallel part. The performance experiment results show that PHN shows the State of the Art on two large benchmark data sets, and explores the interaction layer num of the model. The comparative experimental results show that SSG can effectively improve the representation based on public embedding, and the residual link with trainable parameters can improve the representation ability of the model while maintaining the robustness of the results. Based on the overall experimental results, this work brings us to ont step closer to being able to determine the optimal structure of PHN.