{'fraction_non_alphanumeric': 0.028987284844024696, 'fraction_numerical': 0.01181569156547692, 'mean_word_length': 4.821037601142313, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
FAT-DeepFFM: Field Attentive Deep Field-aware Factorization Machine

Click through rate (CTR) estimation is a fundamental task in personalized advertising and recommender systems. Recent years have witnessed the success of both the deep learning based model and attention mechanism in various tasks in computer vision (CV) and natural language processing (NLP). How to combine the attention mechanism with deep CTR model is a promising direction because it may ensemble the advantages of both sides. Although some CTR model such as Attentional Factorization Machine (AFM) has been proposed to model the weight of second order interaction features, we posit the evaluation of feature importance before explicit feature interaction procedure is also important for CTR prediction tasks because the model can learn to selectively highlight the informative features and suppress less useful ones if the task has many input features. In this paper, we propose a new neural CTR model named Field Attentive Deep Field-aware Factorization Machine (FAT-DeepFFM) by combining the Deep Field-aware Factorization Machine (DeepFFM) with Compose-Excitation network (CENet) field attention mechanism which is proposed by us as an enhanced version of Squeeze-Excitation network (SENet) to highlight the feature importance. We conduct extensive experiments on two real-world datasets and the experiment results show that FAT-DeepFFM achieves the best performance and obtains different improvements over the state-of-the-art methods. We also compare two kinds of attention mechanisms (attention before explicit feature interaction vs. attention after explicit feature interaction) and demonstrate that the former one outperforms the latter one significantly.


Introduction
CTR estimation is a fundamental task in personalized advertising and recommender systems. Many models have been proposed to resolve this problem such as Logistic Regression (LR) [McMahan et al., 2013], Polynomial-2 (Poly2) [Juan et al., 2016], tree-based models [He et al., 2014], tensorbased models [Koren et al., 2009], Bayesian models [Graepel et al., 2010], and Field-aware Factorization Machines (FFMs) [Juan et al., 2016]. Deep learning techniques have shown promising results in many research fields such as computer vision [Krizhevsky et al., 2012;He et al., 2016], speech recognition [Graves et al., 2013] and natural language understanding [Mikolov et al., 2010;Cho et al., 2014]. As a result, employing DNNs for CTR estimation has also been a research trend in this field Cheng et al., 2016;Xiao et al., 2017;Guo et al., 2017;Lian et al., 2018;Wang et al., 2017;He and Chua, 2017]. Some deep learning based models have been introduced and achieved success such as Factorisation-Machine Supported Neural Networks(FNN) , Attentional Factorization Machine (AFM) [Xiao et al., 2017],wide & deep [Cheng et al., 2016],DeepFM [Guo et al., 2017] etc. On the other side, Attention mechanism can filter out the uninformative features from raw inputs and attention-based model has also been widely used and shown promising results on various tasks. How to combine the attention mechanism with deep CTR model is a promising direction because it may ensemble the advantages of both sides. Although some CTR model such as Attentional Factorization Machine [Xiao et al., 2017] (AFM) has been proposed to model the weight of second order interaction features, we think the feature importance evaluation before explicit feature interaction is also important for CTR prediction tasks because the model can learn to selectively highlight the informative features and suppress less useful ones if the task has many input features.
In this work, we propose a new neural CTR model named Field ATtentive Deep Field-aware Factorization Machines (FAT-DeepFFM) by combining the neural field-aware factorization machines [Yang et al., 2017] with field attention mechanism. Specifically, what we do in this work is to introduce the Compose-Excitation network (CENet) which is an enhanced version of Squeeze-Excitation network (SENet) [Hu et al., 2017] like attention into DeepFFM model in order to improve the representational ability of deep CTR network. We aim to dynamically capture each feature's importance by explicitly modeling the interdependencies among all different features of an input instance before factorization machines's feature interaction procedure. Our goal is to use the CENet like attention mechanism to perform feature recali-bration through which it can learn to selectively highlight the informative features and suppress less useful ones effectively.
The contributions of our work are summarized as follows: 1) We propose a novel model named FAT-DeepFFM that enhances the DeepFFM model by introducing the CENet field attention to dynamically capture each feature's importance before explicit feature interaction procedure. 2) We compare two different kinds of attention mechanisms(attention on features before explicit feature interaction vs. attention on cross features after explicit feature interaction ) and the experiment results demonstrate that the former one outperforms the latter one significantly. 3) We conduct extensive experiments on two real-world datasets and the experiment results show that FAT-DeepFFM achieves the best performance and obtains different improvements over the state-of-the-art methods. The rest of this paper is organized as follows. Section 2 introduces some related works which are relevant with our proposed model. We introduce our proposed Field Attentive Deep Field-aware Factorization Machine (FAT-DeepFFM) model in detail in Section 3. The experimental results on Criteo and Avazu datasets are presented and discussed in Section 4. Section 5 concludes our work in this paper.

Factorization Machines and Field-aware Factorization Machine
Factorization Machines (FMs) [Rendle, 2010] and Fieldaware Factorization Machines (FFMs) [Juan et al., 2016] are two of the most successful CTR models. FMs use the dot product of two embedding vectors to model the effect of pairwise feature interactions. FFMs extended the ideas of Factorization Machines by additionally leveraging the field information and won two competitions hosted by Criteo and Avazu. When one feature interacts with other features from different fields, FFMs will learn different embedding vectors for each feature.

Deep Learning based CTR Models
With the great success of deep learning in many research fields such as Computer Vision and Natural language processing, many deep learning based CTR models have also been proposed in recent years. How to effectively model the feature interactions is the key factor for most of these neural network based models.  [Lian et al., 2018] also models the loworder and high-order feature interactions in an explicit way by proposing a novel Compressed Interaction Network (CIN) part.
Our approach is based on neural FFM which was firstly proposed by Yang[Yang et al., 2017] in Tencent Social Ads contest . It can be regarded as replacing the FM part of DeepFM with FFM and we will describe the model in detail in section 3.

Attentive CTR Models
Attention mechanism is motivated by human visual attention and it can filter out the uninformative features from raw inputs by reducing the side effects of noisy data. Attention-based model has been widely used and shown promising results on tasks such as speech recognition and machine translation. Attention mechanism is also introduced in some CTR models. For example, Attentional Factorization Machine (AFM) [Xiao et al., 2017] improves FM by discriminating and learning the importance of different feature interactions from data via a neural attention network. DIN  represents users' diverse interests with an interest distribution and designs an attention-like network structure to locally activate the related interests according to the candidate ad.

DeepFFM
Our work initially aims at introducing the FFM model into neural CTR systems. However, a similar effort to ours has been reported by Yang etc. [Yang et al., 2017] in Tencent Social Ads competition 2017. The authors report substantial gains after using neural FFM in their CTR prediction system. Neural FFM was quite successful in that competition: the 3rd place winner solution was based on this single Model and the ensemble version won the 1rd place in the competition. Because it's hard to find the detailed technical descriptions about this model, we will firstly introduce the neural FFM which will be called DeepFFM model in this paper.
As we all know, FMs [Rendle, 2010] model interactions between features i and j as the dot products of their corresponding embedding vectors as follows: An embedding vector v i ∈ R k for each feature is learned by FM, k is a hyper-parameter which is usually a small integer and m is the feature number. However, FM neglects the fact that a feature might behave differently when it interacts with features from other fields. To explicitly take this difference into consideration, Field-aware Factorization Machines (FFMs) learn extra n-1 embedding vectors for each feature(here n denotes field number): where v ij ∈ R k denotes the embedding vector of the j-th entry of feature i when feature i is interacting with fields j. k is the embedding size. As depicted in Figure 1, DeepFFM is designed to embody the idea of FFM through neural network. An input instance is firstly transformed into a high-dimensional sparse features via one-hot encoding to denote the raw feature input. The following embedding matrix layer is fully connected with sparse input layer to compress a raw feature to a low dimensional, dense real-value matrix. Specifically, for feature i, a corresponding 2-dimensional embedding matrix with size k × n is used to measure its impact of interactions with other features, where v ij ∈ R k refers to the j-th embedding vector of field i, n is the number of fields and k is the size of embedding vector. So it's obvious that embedding matrix layer EM is a 3-dimensional matrix with size k × n × n because we have n fields and each field has one corresponding 2-dimensional embedding matrix.
The following feature interaction layer tries to capture the two way feature interactions between any pair of features from different fields on the embedding matrix EM . Denoting the feature interaction layer as vector A, we have two different types of feature interaction approaches: inner-product version and Hadamard-product version. We can formalize two methods in this layer as follows: where n is field number, v ij ⊕ v ji means the inner product of two embedding vectors as a scalar v ij , v ji and v ij × v ji refers to the Hadamard product of two embedding vectors as following vector: where k is the size of embedding vector v ji . Notice that j > i is required in order to avoid the repeated computation. We can see from here that feature interaction layer A is a wide concatenated vector and the size of this vector is n(n−1)/2 if we adopt inner-product version while the size is kn(n − 1)/2 if the Hadamard product version is adopted.
Multiple hidden layer is a feed-forward neural network on the feature interaction layer to implicitly learn high-order feature interactions. Denote the output of the feature interaction layer as vector and we can feed it into hidden layer of feedforward neural network. So the forward process is : where l is the layer depth, σ is an activation function, and x l is the output of the l-th hidden layer.
Adding the linear part, the output unit of DeepFFM behaves as follows: where σ is the sigmoid function, x linear is the raw features, x l is the output of multiple hidden layer, W 1inear , W l+1 and b l+1 are learnable parameters.

CENet Field Attention on Embedding Matrix Layer
Hu proposed the "Squeeze-and-Excitation Network" (SENet) [Hu et al., 2017] to improve the representational power of a network by explicitly modeling the interdependencies between the channels of convolutional features in various image classification tasks. The SENet is proved to be successful in image classification tasks and won first place in ILSVRC 2017 classification task. Our work is inspired by SENet's success in computer vision field. To improve the representational ability of deep CTR network, we introduce the Compose-Excitation network (CENet) attention mechanism which is an enhanced version of SENet into DeepFFM model on embedding matrix Layer. We aim to dynamically capture each feature's importance by explicitly modeling the interdependencies among all different features before FM's feature interaction procedure. Our goal is to use the CENet attention mechanism to perform feature recalibration through which it can learn to selectively highlights the informative features and suppress less useful ones.
It can be seen from Figure 2 that the CENet like field attention mechanism involves two phases: Compose phase and Excitation phase. The first phase calculates "summary statistics" of each embedding vector of each field by composing all the information of one embedding vector into a simple feature descriptor; the second phase applies attentive transformations to these feature descriptors and then rescales the original embedding matrix using the calculated attention values.
where v ij ∈ R k refers to the j-th embedding vector of field i, n is the number of fields and k is the size of embedding vector. In this phase we compose the embedding vector v ij into one single number to represent the summary information of the feature. This can be achieved by using 1 × 1 convolution [Szegedy et al., 2016;Chollet, 2017] to generate feature-wise statistics instead of those squeeze operations such as global max pooling or sum operation commonly used in SENet. The 1 × 1 convolution, also called a pointwise convolution, is responsible for building new features through computing linear combinations of one input feature embedding. In SENet, we generate a statistic vector z ∈ R n for field i by shrinking each embedding vector, where the f-th element of z i is z if ∈ R which can be calculated by: Global M ax P ooling (6) Here, k means the embedding size of each embedding vector.
The most commonly used squeeze operation is global max pooling in CV field which can capture the strongest feature in corresponding channel. We change the method in this phase by using 1 × 1 convolution because we posit each position in feature embedding vector is informative in CTR task. So the 1 × 1 convolution can introduce parameters to learn the composing weight of each position in feature embedding. The 1 × 1 convolution is calculated as follows: where U if is the convolution weight, the size of convolution kernal is 1 × 1, the number of filters is 1 and the activation function is set to 'Relu'. Excitation phase: After the first phase, the embedding matrix of field i EM i = [v i1 , v i2 , · · · , v ij , · · · , v in ] has been transformed into a descriptor vector DV i = [z i1 , z i2 , · · · , z ij , · · · , z in ] . We have n different fields, so we summarize all the descriptors by concatenating each descriptor vector as follows: where the size of vector D is n 2 . To calculate the attention from descriptor vector , two fully connected (FC) layers are used. The first FC layer is a dimensionality-reduction layer with parameters W 1 with reduction ratio r which is a hyper-parameter and it uses ReLU as nonlinear function. The second FC layer increases dimensionality with parameters W 2 , which is equal to dimension of descriptor vector D and it also uses ReLU as nonlinear activation function. Formally, the field attention is calculated as follows: where δ refers to the ReLU function, W 1 ∈ R n 2 r ×n 2 and W 1 ∈ R n 2 × n 2 r ,the size of attention vector S is n 2 . The activation of the ReLU function is used as the final field attention value without softmax normalization operation because we want to encourage multiple features to be important instead of just few of them. Then the values in original embedding matrix EM i of field i are rescaled by the accordingly calculated field attention vector S i as follows: (10) where F scale (S i , EM i ) refers to vector-wise multiplication between embedding vector v ij and the scalar S ij . The bigger attention value S ij implies that the model dynamically identifies an important feature and this attention value is used to boost the original embedding vector v ij . On the contrary, small attention value S ij will suppress the uninformative features or even noise by decreasing the values in the corresponding embedding vector v ij .
After the compose phase and excitation phase, we have a new 3-dimensional embedding matrix AEM with size k ×n×n, which is equal to the size of the original embedding matrix EM . We call the new embedding matrix attentive embedding layer in our paper. As discussed in Section 3.2, the CENet attention mechanism can perform feature recalibration through which it can learn to selectively highlights the informative features and suppress less useful ones. We can enhance DeepFFM model which is described in section 3.1 by inserting the CENet attention module into it. Figure 3 provides overall architecture of our proposed Field Attentive Deep Field-aware Factorization Machine (FAT-DeepFFM). It's similar in neural structure to DeepFFM while the original embedding matrix layer is replaced by the SE-Net like field attention module. We call this newly plugged-in module attentive embedding matrix layer. The other components of FAT-DeepFFM are same as the DeepFFM model. Similar to the DeepFFM, there are also two versions of FAT-DeepFFM according to the feature interaction type: inner-product version and Hadamard-product version.

Combining the field attention and DeepFFM
We can see from the above-mentioned descriptions that our proposed attention mechanism is a kind of attention before cross features were produced. So a natural research question arises that which one will perform better if we introduce attention on cross features after the explicit feature interaction procedure just like AFM does? To answer this question, we also conduct some experiments to compare the performance difference of two kinds of attention mechanisms. The experimental results demonstrate that the attention before feature interaction outperforms the one after feature interaction consistently. We will discuss these experiments in detail in Section 4.3.

Experimental Results
To comprehensively evaluate our proposed method, we design some experiments to answer the following research questions: RQ1 Can our proposed FAT-DeepFFM outperform the stateof-the-art deep learning based CTR models? RQ2 Which attention mechanism (attention on features before explicit feature interaction vs. attention on cross features after explicit feature interaction) will perform better on the real world CTR datasets? RQ3 Which feature interaction method (Inner-Product vs. Hadamard-product) is more effective in neural network based CTR models?

Experiment Setup Datasets
The following two data sets are used in our experiments: 1. Criteo 1 Dataset. As a very famous public real world display ad dataset with each ad display information and corresponding user click feedback, Criteo data set is widely used in many CTR model evaluation. There are 26 anonymous categorical fields and 13 continuous feature fields in Criteo data set. We split the data into training and test set randomly by 90%:10%. 2. Avazu 2 Dataset. The Avazu dataset consists of several days of ad click-through data which is ordered chronologically. For each click data, there are 24 fields which 1 http://labs.criteo.com/downloads/ 2 http://www.kaggle.com/c/avazu-ctr-prediction indicate elements of a single ad impression. We split the data into training and test set randomly by 80%:20%. Table 1 lists the statistics of the evaluation datasets. For these two datasets, a small improvement in prediction accuracy is regarded as practically significant because it will bring a large increase in a company's revenue if the company has a very large user base.
Evaluation Metrics AUC (Area Under ROC) and Logloss (cross entropy) are used in our experiments as the evaluation metrics. These two metrics are very popular for binary classification tasks. AUC is insensitive to the classification threshold and the positive ratio. AUC's upper bound is 1 and larger value indicates a better performance. Log loss measures the distance between two distributions and smaller log loss value means a better performance.

Models for Comparisons
We compare the performance of the following CTR estimation models as baseline:LR, FM, FFM, FNN, DeepFM, AFM, Deep&Cross Network(DCN) , xDeepFM and DeepFFM, all of which are discussed in Section 2 and Section 3.

Implementation Details
We implement all the models with Tensorflow in our experiments. For optimization method, we use the Adam with a mini-batch size of 1000 and a learning rate is set to 0.0001. Focusing on neural networks structures in our paper, we make the dimension of field embedding for all models to be a fixed value of 10. For models with DNN part, the depth of hidden layers is set to 3, the number of neurons per layer is 1600 for FFM-related models and 400 for all other deep models, all activation function are ReLU and dropout rate is set to 0.5. For CENet component, the activation function is ReLU and the reduction ratio is set to 1 in all the related experiments. We conduct our experiments with 2 Tesla K40 GPUs.

Performance Comparison (RQ1)
The overall performance for CTR prediction of different models on Criteo dataset and Avazu dataset is shown in Table  2. We have the following key observations: 1. FAT-DeepFFM achieves the best performance in general and obtains different improvements over the state-ofthe-art methods. As the best model, FAT-DeepFM outperforms FM by 3.64% and 1.80% in terms of Logloss (2.28% and 1.50% in terms of AUC) and outperforms LR by 5.64% and 3.29% in terms of Logloss (3.79% and 2.99% in terms of AUC) on Criteo and Avazu datasets. 2. FAT-DeepFFM consistently outperforms DeepFFM on both datasets. This indicates that CENet field attentive mechanism is rather helpful for learning the importance of raw features.

Attention Mechanism Comparison (RQ2)
In this subsection, we will discuss the performance of two different kinds of attention mechanisms: one is an attention before explicit feature interaction just like above-mentioned field attention; the other is the attention on cross features after explicit feature interaction procedure just like AFM does. As for the specific approach for the attention on cross features, two methods are implemented: the similar CENet attention as described in section 3.2 and the MLP based attention just like AFM does, the hyper-parameters are tuned to achieve the best performance. The experimental results is shown in table 3 and the experiments with prefix "MLP" refer to the MLP based attention on cross features while the experiments with prefix "CE" mean the CENet attention is used on cross features. Table 3 lists the overall performance of two attention mechanisms on Criteo dataset and Avazu dataset. We have the following key observations:  As we discussed in section 3, both the DeepFFM and FAT-DeepFFM model have two kinds of feature interaction approaches: inner product version vs. hadamard product version. So a natural research question arises that which approach will perform better?

Conclusion
In this paper, we propose a new neural CTR model called Field Attentive Deep Field-aware Factorization Machine (FAT-DeepFFM) by combining the deep field-aware factorization machine (DeepFFM) with CENet field attention mechanism. We conduct extensive experiments on two realworld datasets and the experiment results show that FAT-DeepFFM achieves the best performance and obtains different improvements over the state-of-the-art methods. We also show that FAT-DeepFFM consistently outperforms DeepFFM on both datasets which indicates that CENet field attentive mechanism is rather helpful for learning the importance of raw features when the task has many input features. We also compare two different types of attention mechanisms (attention before explicit feature interaction vs. attention after explicit feature interaction) and the experiment results demonstrate that the former one outperforms the latter one significantly.