{'fraction_non_alphanumeric': 0.031291772997934134, 'fraction_numerical': 0.0077773727062826585, 'mean_word_length': 5.031146940271162, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards a Better Tradeoff between Effectiveness and Efficiency in Pre-Ranking: A Learnable Feature Selection based Approach

In real-world search, recommendation, and advertising systems, the multi-stage ranking architecture is commonly adopted. Such architecture usually consists of matching, pre-ranking, ranking, and re-ranking stages. In the pre-ranking stage, vector-product based models with representation-focused architecture are commonly adopted to account for system efficiency. However, it brings a significant loss to the effectiveness of the system. In this paper, a novel pre-ranking approach is proposed which supports complicated models with interaction-focused architecture. It achieves a better tradeoff between effectiveness and efficiency by utilizing the proposed learnable Feature Selection method based on feature Complexity and variational Dropout (FSCD). Evaluations in a real-world e-commerce sponsored search system for a search engine demonstrate that utilizing the proposed pre-ranking, the effectiveness of the system is significantly improved. Moreover, compared to the systems with conventional pre-ranking models, an identical amount of computational resource is consumed.


INTRODUCTION
As important internet services, large-scale search engine and recommendation systems play important roles in information retrieval and item recommendation, where a ranking system selects only a few items from tens of millions of candidates. Under the constraint of extremely low system latency, a single complicated ranking model cannot rank the entire candidate set. Therefore, multistage ranking architecture is commonly adopted [2,7,9] (shown in Figure 1 (a)). Large-scale deep neural networks (DNNs) with interaction-focused architecture [3] are usually employed for the ranking model to maintain good system performance, while only less complicated models with representation-focused architecture [3] are adopted in pre-ranking to ensure efficiency.
However, simple pre-ranking models with representation-focused architecture will inevitably diminish the model expression ability. Vector-product based model [4,13], which is classified as representationfocused architecture, is often employed in pre-ranking (see Figure 1 (b)). The limitation is that neither explicit interactive features nor implicit interactive semantics, which are less efficient for computation but very effective for the expression ability [11], can be used. The pre-ranking models with representation-focused architecture focus excessively on the efficiency optimization and remain a large effectiveness gap to models with interaction-focused architecture. Moreover, as the online feature generation process consume as much resource as that needed for the online inference of the model, the utilization for the pre-ranking with interaction-focused architecture is possible by using feature selection considering both effectiveness and efficiency. In this context, the pre-ranking with an interaction-focused architecture is studied in this paper.
In this paper, a pre-ranking model with tradeoff of both effectiveness and efficiency is proposed for a real-world e-commerce application, where the effectiveness improves significantly with slight decrease of efficiency shown in Figure 1 (b). The contributions of this paper are summarized as follows. 1) A pre-ranking model with interaction-focused architecture is proposed by inheriting the architecture of ranking model to solve the problem of performance loss of the model with representation-focused architecture. 2) A learnable Feature Selection method based on feature Complexity and variational Dropout (FSCD) is proposed to search for a set of effective and efficient feature fields for the pre-ranking model. 3) Extensive experiments are carried out which show the proposed approach achieves great improvement in effectiveness of the system compared to the conventional baselines, while the efficiency of the system is also maintained. Multi-stage architecture including matching, pre-ranking, ranking, and re-ranking with scoring item numbers. (b) Intuitive view for effectiveness and efficiency of pre-ranking and ranking, where the pre-ranking is derived from ranking and optimized to an interaction-focused architecture.

THE PROPOSED APPROACH
The proposed pre-ranking is derived from the ranking model. Both models utilize interaction-focused architectures, which shares an identical feature set = { 1 , 2 , ..., }, where is the -th feature field in , and is the number of feature fields. In the system, the ranking model employs all feature fields in to maintain effectiveness, while the pre-ranking model utilizes only a subset to reduce computational complexity and scores more items. In this context, offline processes such as sample generation can be reused for both models, which saves offline computational resources. The interaction-focused architecture of pre-ranking is inherited from that of ranking model, where both explicit interactive features and implicit interactive semantics can be utilized, which reduces the gap in the models' optimization objectives and achieves significant improvement of the model effectiveness for pre-ranking compared to the model with representation-focused architecture.
For introducing interaction-focused architecture into pre-ranking, a great challenge is the model efficiency. The overall efficiency of the pre-ranking model, whose learnable variables can be divided into feature embeddings and dense weights , is strongly influenced by the feature embeddings . For example, in real-world scenarios, storage for feature embeddings exceeds over 95% of that for the whole model, while feature generation process for embeddings consumes as much resources as that needed for the online inference of the model. Therefore, the feature selection for features with both high effectiveness and efficiency is important for the utilization of pre-ranking with interaction-focused architecture.

FSCD for Pre-Ranking Model
Inspired by the Dropout FR method based on the variational dropout layer [1] where the efficiency of the model is ignored, FSCD is proposed that considers both effectiveness and efficiency in a learnable process as illustrated in Figure 2. In the proposed FSCD, the effectiveness is optimized by the cross entropy based loss function, while the efficiency is optimized by the feature-wise regularization term in Eq. (3). Both effective and efficient features can be selected by FSCD in one single training process, while the expression ability of pre-ranking model is improved utilizing these features compared to the vector-product based model. The details are as follows. To select feature fields with both high effectiveness and high efficiency, each feature field is expected to learn a dropout parameter ∈ {0, 1} to indicate whether the feature field is dropped ( = 0) or preserved ( = 1). 's embeddings are multiplied by the dropout to form the embedding layer, and is subject to a Bernoulli distribution parameterized by , i.e., where the hyperparameter is the priori probability for the preservation of feature field and is configured as function of feature complexity , i.e., where (·) is the sigmoid function. = 1 − ( ) is one of alternatives to relate and which works well in practice. The feature complexity measures the computational and storage complexity of the -th feature field including but not limited to the online computational complexity , the embedding dimension , and the number of keys for one feature field, where is configured according to the feature type specified in Section 3.1. Linear combination = 1 + 2 + 3 is one of the alternatives with hyperparameter 1,2,3 . According to Eq. (2), a feature with large complexity has a small value of , and vice versa.
Given the training samples D = {( , )| = 1, 2, ..., }, where is the number of samples, the overall loss function for the learnable feature selection is derived by Bayesian rule [10] as, where is the regularization weight for and can be derived as (See derivation details in Appendix A) is a function that decreases with and increases with . Therefore, a feature with larger complexity is penalized with a larger value of , and more likely to be dropped. In this way, the feature complexity is included in the proposed FSCD, which previous works do not address to the best of the authors' knowledge.
Subject to Bernoulli distribution, is discrete and not differentiable, it is relaxed to a differentiable function as [5] where ∼ Uniform(0, 1) is subject to a uniform distribution and changes during the training process, while = 0.1 is a constant that works well in the experiments. F ( ) is close to 0 or 1 for most values, which approximates the discrete Bernoulli distribution. In contrast to the dropout , ∈ (0, 1) is a differentiable parameter. Moreover, it acts as the posterior probability for feature preservation, which is influenced by the priori probability for feature preservation , and can be learned as the feature importance.
In this context, the entire process of training for feature selection is built in Eq. (3) and (5). Note that, the learnable variables are , and , which are trained simultaneously. Through a fast convergence of , the feature set of the pre-ranking model can then be obtained by selecting the feature fields with top-of values.

Fine-Tuning the Pre-Ranking Model
After feature selection, the feature fields not in the pre-ranking set acquired by FSCD are masked, and the models are fine-tuned using weights and as initialization parameters. Concretely, the model can be trained with the following loss function where ′ , ′ , and ′ are the samples with the selected feature fields, the remaining feature embeddings, and dense weights for the preranking model, respectively. ′ and ′ are initialized by and , which accelerates the training. The Bernoulli dropout is omitted in the fine-tuning process.
In this way, the pre-ranking model is obtained. As the model is in an interaction-focused architecture and adopts both effective and efficient feature fields, its expression ability significantly improves. Therefore, the effectiveness of the system can be improved with high efficiency. The entire method is illustrated in Algorithm 1.

EXPERIMENTS 3.1 Experiment Configurations
Evaluations are mainly based on the online sponsored search system for a real-world e-commerce app named Taobao, where both the click-through rate (CTR) and response per mile (RPM), which evaluates platform revenue, are optimized. The proposed effective and efficient pre-ranking are compared with the baselines of the vector-product based model [4] and COLD [12] based pre-ranking.
Both the pre-ranking and ranking models are based on an interactionfocused architecture. After the feature fields are transformed into embeddings, hidden layers with sizes of 1024, 512, and 256 are adopted for the ranking, while only 2 hidden layers with sizes of 1024 and 256 are used for the pre-ranking to ensure high efficiency. Finally, the sigmoid function is utilized to predict the final CTRs for both models.
The feature set for the pre-ranking model is a subset of that for the ranking model, which is selected by the proposed FSCD. All feature sets, consisting of 246 feature fields in their entirety,

Analysis of FSCD
After the proposed FSCD process, the feature fields are ranked using the output parameter . Table 2 lists the minimum, median, and maximum ranking indices for different types of features using Dropout FR [1] and deep feature selection (DFS) method [6]. For the proposed FSCD in this paper, the ranking indices for the simple features are much smaller than those for the complex feature types (see median and maximum). The simple features tend to have a more forward ranking, while the complicated features are backward, which emphasizes the efficiency of the features. However, the minimum ranking index of type IV features is only 8, which means that a feature with large complexity can still rank forward if it is truly effective for the training task. For the conventional methods, the rankings of the features are less dependent on the feature type. The pre-ranking models with different feature field number are well trained based on Eq. (6) with 2 billion samples. The area under the curve (AUC), latency, and CPU consumption for online inference of the different models are illustrated in Table 3. The results of the conventional feature selection methods [1,6] are also provided for comparison. Specifically, when = 246, the feature set of pre-ranking model becomes exactly the same as that of the ranking model. The results show that the proposed FSCD has slightly smaller AUC than the other methods when identical values are considered. However, the complexity is extremely   Table 5: Online effectiveness and efficiency of the proposed pre-ranking model and the conventional models on realworld mobile online queries.
reduced. When = 100, the AUC difference relative to that of the model for the other methods is only 0.0026, while the complexity is approximately 30% lower than that of the other methods for both latency and CPU cost. When > 100, the AUC increases slowly, while the complexity increases significantly. Therefore, the pre-ranking utilizes the feature fields that have top-100 of values as the final feature set. Table 4 and Table 5 show the offline and online experimental results using the proposed effectiveness and efficiency based pre-ranking, while the original vector-product and COLD based pre-rankings are configured as benchmarks. Each model is trained with more than 200 billion real-world e-commerce samples from a log system to achieve the best online performance. Therefore, the offline AUCs are much larger than those shown in Table 3. The recall rate is defined as the preserved probability of pre-ranking model for the top-5 items ranked by the ranking model. The offline results in Table 4 show that the proposed model is much more effective than the vector-product based model, while it is slightly less effective than COLD due to efficiency considerations. For the online effect, 30 continuous days of mobile real-world requests are evaluated and the results are shown in Table 5. It shows that the proposed pre-ranking model obtains a good balance for online CTR and RPM performance and gains significant improvement in both CTR and RPM compared with its counterparts.

Performance Comparisons
Finally, the efficiency is analyzed. Although a complicated preranking with an interaction-focused architecture is adopted in the proposed approach, the feature selection method based on effectiveness and efficiency optimizes the computational complexity to a low degree. Moreover, the number of input item is reduced from 6000 to 800, which further reduces the overall complexity of the system. In this context, the online CPU consumption of the system is almost the same as that of the vector-product based model, while the response time is slightly increased. In regard to the COLD model, the response time and CPU consumption are both greater than those of the proposed approach, which means the proposed pre-ranking model is more efficient. The key metrics for efficiency are listed in Table 5 at a peak number of queries per second (QPS).

CONCLUSIONS
In this paper, to solve the problem of performance loss caused by the pre-ranking model with representation-focused architecture, a pre-ranking model based on feature selection with joint optimization for both effectiveness and efficiency is proposed in an interaction-focused architecture. The pre-ranking model with interaction-focused architecture, which is inherited from that of ranking model, utilizes the feature subset selected by the proposed learnable FSCD, which includes not only simple features but also interactive features with significant effectiveness. The experiments on offline training demonstrate the validity of the proposed FSCD method in both effectiveness and efficiency. Moreover, the offline and online effects in the real-world sponsored search system illustrate the performance improvements in the proposed pre-ranking model compared to the conventional benchmarks. The proposed pre-ranking has been utilized as an online running model for a real-world sponsored search system and has generated substantial revenue for the company.
Then the total loss function can be written as Eq. (3).