{'fraction_non_alphanumeric': 0.038047751804989635, 'fraction_numerical': 0.01858603188219315, 'mean_word_length': 4.833715596330276, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
AMEIR: Automatic Behavior Modeling, Interaction Exploration and MLP Investigation in the Recommender System

Recently, deep learning models have been widely spread in the industrial recommender systems and boosted the recommendation quality. Though having achieved remarkable success, the design of task-aware recommender systems usually requires manual feature engineering and architecture engineering from domain experts. To relieve those human efforts, we explore the potential of neural architecture search (NAS) and introduce AMEIR for Automatic behavior Modeling, interaction Exploration and multi-layer perceptron (MLP) Investigation in the Recommender system. The core contributions of AMEIR are the three-stage search space and the tailored three-step searching pipeline. Specifically, AMEIR divides the complete recommendation models into three stages of behavior modeling, interaction exploration, MLP aggregation, and introduces a novel search space containing three tailored subspaces that cover most of the existing methods and thus allow for searching better models. To find the ideal architecture efficiently and effectively, AMEIR realizes the one-shot random search in recommendation progressively on the three stages and assembles the search results as the final outcome. Further analysis reveals that AMEIR's search space could cover most of the representative recommendation models, which demonstrates the universality of our design. The extensive experiments over various scenarios reveal that AMEIR outperforms competitive baselines of elaborate manual design and leading algorithmic complex NAS methods with lower model complexity and comparable time cost, indicating efficacy, efficiency and robustness of the proposed method.


Introduction
Recommender system has become an essential service on online E-commerce business and content platforms to deliver items that best fit users' interests from the substantial number * Equal Contribution of candidates. Recently, advancements of deep learning (DL) have innovated the recommending strategies in real-world applications [Covington et al., 2016;Zhou et al., 2018b] to enable accurate and personalized recommendation.
DL-based recommender systems have to process both sequential and non-sequential input features, and thereby follow a characterized architecture compared to the general DL tasks. A canonical DL-based recommendation model usually includes three parts: (1) Behavior modeling [Hidasi and Karatzoglou, 2018;Kang and McAuley, 2018;Tang and Wang, 2018;Zhou et al., 2018b] probes into capturing user's diverse and dynamic interest from historical behaviors; (2) Interaction exploration [Guo et al., 2017;Lian et al., 2018;Qu et al., 2018;Wang et al., 2017] finds useful interactions among different fields to provide memorization and intuitive conjunction evidence for the recommendation. (3) Multilayer perceptron (MLP) investigation [Covington et al., 2016;Zhou et al., 2018b] aggregates the inputs with the results from behavior modeling and interaction exploration, and then fuses the features with hidden layers in the MLP.
Industry demands efficiently designing the three parts in the DL-based recommendation model to improve the accuracy and personalization for given recommender tasks in business. Nevertheless, it is hard to find a unified model meeting the requirements in all the scenarios. For the models of behavior modeling, recurrent neural network (RNN) [Hidasi and Karatzoglou, 2018;Hidasi et al., 2016] is hard to preserve long-term behavioral dependencies even though employing gated memory cells. Convolutional neural network (CNN) [Tang and Wang, 2018;Yuan et al., 2019] is capable of learning local feature combinations yet it relies on a wide kernel, deep network or global pooling to distinguish long-term interests. The attention mechanism [Zhou et al., 2018b] directly aggregates the entire behavior sequence, but it can not capture the evolution of the user's preference [Zhou et al., 2019]. Self-attention [Vaswani et al., 2017] is better for modeling long-term behaviors [Feng et al., 2019;Sun et al., 2019], but it is hard to be deployed in realtime applications that require fast inference with limited resources. Regarding the methods for interaction exploration and MLP investigation, vanilla MLP [Covington et al., 2016] implicitly generalizes high-order information in the network whereas missing the low-order information. Therefore, linear regression (LR) [Cheng et al., 2016], factoriza-tion machine (FM) [Guo et al., 2017;He and Chua, 2017] and product-based neural network (PNN) [Qu et al., 2016;Qu et al., 2018] are introduced to involve low-order feature interactions from raw input categorical features. On top of that, DCN  and xDeepFM [Lian et al., 2018] further learn compressed high-order interactions in the networks. However, these methods either require handcrafted cross features or simply enumerate interactions of bounded degree, and thus introducing noise in the model [Liu et al., 2020]. Moreover, the MLP specified manually or by grid search is usually sub-optimal. Therefore, adopting and coordinating the task-aware architectures for the three parts of recommendation models are the main challenge for building efficient and accurate recommender systems.
Recently, the neural architecture search (NAS) paradigm [Zoph et al., 2018;Xie et al., 2019;Real et al., 2019;Tan and Le, 2019;So et al., 2019;Wang et al., 2020b] is proposed to automatically design deep learning models by searching task-specific and data-oriented optimal architecture from the search space, thereby mitigating a lot of human efforts. Some latest attempts [Luo et al., 2019;Liu et al., 2020;Song et al., 2020] incorporate NAS with the recommendation.
Nevertheless, they all neglect behavior modeling and only consider interaction exploration and MLP investigation in the restricted search spaces. Different from these methods, we introduce AMEIR, namely Automatic behavior Modeling, interaction Exploration and MLP Investigation in the Recommender system, to automatically find the complete recommendation models for adapting various recommender tasks and mitigating manual efforts. Specifically, AMEIR divides the backbone architecture of recommendation models based on the three stages of behavior modeling, interaction exploration, MLP aggregation, and introduces a novel search space containing three tailored subspaces that are designed to cover most of the representative recommender systems and thus allow for searching better architectures. Facing the industrial demands of agile development and architecture iteration, AMEIR realizes the efficient while competitive one-shot random search [Li and Talwalkar, 2019] in this paper, and proposes a three-step searching pipeline to progressively find the ideal architectures for the corresponding three stages in the recommendation model. The search results will be assembled as the final outcome. AMEIR would not add complicated or redundant operations to the computation graph, and thus the yielded models can be optimized by the common industrial accelerating strategies [NVIDIA, 2020]. The experimental results over various recommendation scenarios show that our automatic paradigms consistently achieves state-of-the-art performance despite the influence of multiple runs, and could outperform both the strongest baselines with elaborate manual design and leading NAS methods with comparable search cost, indicating efficacy, efficiency, and robustness of the proposed method. Further analysis reveals that the overall time cost of AMEIR is similar to the baseline methods without architecture search and the searched models always possess lower model complexity, which demonstrates that AMEIR is efficient in both search and inference phases.

AMEIR
In Sec.2, we will introduce the three-stage search space, and the tailored three-step one-shot searching pipeline in AMEIR. We will also discuss the relationship between AMEIR and representative recommender systems and show that AMEIR is able to cover most of these methods.

Backbone Model in AMEIR's Search Space
The recommendation models in AMEIR's search space share the same three-stage backbone, as illustrated in Fig.1. Stage 0: Feature Representation and Embedding. In the common recommender systems, the input features are collected in multi-field categorical form, and Generally, there are three basic elements in the data instances: user profile, item profile and context profile. User profile represents the user's personal information and can be expressed by a tuple of multiple fields, e.g., ("User ID", "User Attributes", etc). Analogously, item profile and context profile can be represented by ("Item ID", "Item Attributes", etc) and ("Time", "Place", etc). Based on the basic elements, AMEIR segregates the features into two groups: segregated into sequential features and non-sequential features: (1) The sequential features describe the user's historical behaviors, which can be represented by a sequence of behavior elements. Each behavior element contains the corresponding item profile and the optional context profile. (2) The non-sequential features depict the attribute information of the current recommendation, including user profile, instant context profile, and the optional target profile for click-through rate (CTR) prediction task.
Conforming to [Covington et al., 2016;Zhou et al., 2018b], AMEIR transforms the data instances into highdimensional sparse vectors via one-hot or multi-hot encoding. Specifically, if the field is univalent, it will be converted to one-hot vector; if the field is multivalent, it will be encoded as a multi-hot vector. Each data instance can be formally illustrated as: where x a = [x a 1 , x a 2 , ..., x a T ] denotes the encoded sequential behavior elements of length T with each behavior element x a i = [x a i,1 , x a i,2 , ..., x a i,Na ] grouped in N a categorical fields of the same dimension, and represents N encoded vectors of non-sequential feature fields. AMEIR compresses the sparse features into low dimensional dense vectors via embedding technique. Suppose there are F unique features. AMEIR creates the embedding matrix E ∈ R F ×K , where each embedding vector has dimension K. The embedding process follows the embedding lookup mechanism: If the field is univalent, the field embedding is the feature embedding; If the field is multivalent, the field embedding is the sum of feature embedding. The embedded data instance can then be represented by: where e a i = [e a i,1 , e a i,2 , ..., e a i,Na ] ∈ R NaK is the concatenation of the embeddings in x a i = [x a i,1 , x a i,2 , ..., x a i,Na ], and e b i ∈ R K is the embedding of x b i .  Figure 1: The three-stage search space and the three-step one-shot searching pipeline of AMEIR. AMEIR searches for building blocks, feature interactions, and aggregation MLP (three layers as an example) from corresponding search subspaces in Steps 1-3.
Stage 1: Behavior Modeling on Sequential Features. To extract the sequential representation from user's behavior, AMEIR introduces a block-wise behavior modeling network to model sequential patterns based on grouped behavioral embedding H a 0 = [e a 1 , e a 2 , ..., e a T ]. The behavior modeling network is constructed by stacking a fixed number of L a residual building blocks, where the l-th block receives the hidden state H a l−1 from the previous block as input and generates a new hidden state H a l of the same dimension. Each block consists of three consecutive operations of normalization, layer, and activation, which are selected from the corresponding operation sets in the search subspace of building blocks. The hidden state of layer l is formulated as: where Act a l , Layer a l and Norm a l are the selected activation, layer and normalization operations for layer l. As the fully connected networks can only handle fixed-length inputs, AMEIR compresses the final hidden states into fixed dimensions via sequence-level sum pooling: La,i denotes the i-th element in H a La . Moreover, to extract the latest interest of the user, the last hidden state h a La,T is also included in the result. Besides, when the target item is involved, AMEIR adopts an attention layer to extract a target-aware representation: h a att = T i=1 a i h a La,i , with a i computed by the local activation unit [Zhou et al., 2018b]. The concatenation h a = [h a pool , h a La,T , h a att ] is served as the sequential feature extracted from Stage 1. In practice, this stage remains the procedure to execute looking up embedding only once, which is orthogonal to the industrial large-scale sparse-features lookup optimizations [Oldridge et al., 2020]. Stage 2: Interaction Exploration on Non-sequential Features. Learning effective feature interactions is crucial for recommender systems. Different from the widely-used methods that manually extract useful interactions or exhaustively enumerate interactions of bounded degrees [Cheng et al., 2016;Guo et al., 2017;Lian et al., 2018], AMEIR explores a small number of M beneficial interactions p b = [p b 1 , p b 2 , ..., p b M ] from the search subspace of interactions. The search subspace contains all low-order and high-order interactions among the non-sequential features, where each interaction has the same dimension as the input embeddings, i.e., p b i ∈ R K , i ∈ [1, 2, ..., M ]. Stage 3: Aggregation MLP Investigation with Shortcut Connection. Given the inputs of sequential features, non-sequential features, and explored interactions h c 0 = [h a , e b , p b ], AMEIR jointly learns an MLP with dense Squeeze-and-Excitation (SE) [Hu et al., 2018] connection to aggregate these features by taking advantages of memorization from wide shortcut connection and generalization from deep neural network. Assume the aggregation has L c hidden layers. Considering both recommendation quality and efficiency, AMEIR searches for the dimension and activation function for each hidden layer from the search subspace of MLP. The forward pass of each hidden layer l ∈ [1, 2, ..., L c ] can be depicted as: where h c l , W c l , b c l and Act c l are the output, weight, bias and activation function of the l-th layer. Meanwhile, inspired by the bypaths in [Cheng et al., 2016;Guo et al., 2017;Lian et al., 2018], AMEIR introduces a dense connection to help memorize the low-order information from the feature embeddings and explored interactions As the dimension of Q se is several times higher than h c Lc , AMEIR employs the SE shortcut connection with 1.0 expansion ratio for attentive recalibration and feature compression [Liu et al., 2020]. The attention units a se = [a se 1 , a se 2 , ..., a se M +N ] in the SE network are computed by: where v se is applied for linear projection, and W se 1 , W se 2 are weight matrices in the fully-connected layers. The output of SE connection is given by the weighted sum a se i q se i , where q se i is the i-th element in Q se . AMEIR concatenates the results from aggregation MLP and SE connection as the output representation h c = [h c Lc , h se ]. Loss Function. For the CTR prediction task, AMEIR mini-mizes the binary log loss for optimization: where σ(w T h c ) is the predicted CTR score, and S is the training set with x as input and y ∈ {0, 1} as label. For the item retrieval task, AMEIR adopts the sampled binary cross entropy loss as the objective function: where σ(e T i Wh c ) is the relevance score between target item i and final representation; S is the training set of input feature x and target item i; I − i is the item set of negative samples.

Search Subspaces of Three Stages
For each stage in the backbone, AMEIR introduces a corresponding search subspace to cover the representative recommendation models. We will describe each of the three subspaces below and show a detailed comparison in Sec.  et al., 2017], Identity} that are commonly used in the previous methods. Regarding the layer operations, AMEIR introduces four categories of candidate layers, namely convolutional layers, recurrent layers, pooling layers, and attention layers to identify sequential patterns in the user history. The convolutional layers are all onedimension, including standard convolution with kernel size {1, 3} and dilated convolution with kernel size {3, 5, 7}. The pooling layers consist of average pooling and max pooling with kernel size 3. Both convolutional and pooling layers are of stride 1 and SAME padding. Bi-directional GRU (Bi-GRU) is employed as the recurrent layer as it is faster than Bi-LSTM without loss of precision. Two-head and four-head bi-directional self attentions [Vaswani et al., 2017] are also introduced for better behavior modeling. Additionally, when the target item appears in the input, the layer operation set will associate an attention layer that attends to each sequence position from target [Zhou et al., 2018b]. Besides, zero operation is also included to implicitly allow a dynamic depth of behavior modeling network by dropping redundant layers. It is worth emphasizing that the operation sets are proposed to cover the existing methods. When a new operation is developed, it can be easily added to the corresponding operation set to help find a better recommendation model.

Three-step One-shot Searching Pipeline
One-shot NAS in AMEIR. Recent approaches [Bender et al., 2018;Pham et al., 2018] introduce efficient one-shot weight-sharing paradigm in NAS to boost search efficiency, where all child models share the weights of common operations in a large one-shot model that subsumes every possible architecture in the search space. Though diverse complex search algorithms [Cai et al., 2019b;Chen et al., 2019b;Chu et al., 2019;Dong and Yang, 2019b] have been coupled with one-shot NAS, the latest studies [Bender et al., 2018;Dong and Yang, 2020;Li and Talwalkar, 2019] reveal that the efficient one-shot random search is surprisingly better than the complex gradient-based methods of DARTS , SNAS [Xie et al., 2019] and RL-based methods of ENAS [Pham et al., 2018]. To facilitate the industrial demands of agile development and architecture iteration, AMEIR realizes the one-shot random search in the recommendation model: (1) The child models are randomly sampled from the search spaces (actually subspaces) to train the shared weights of the one-shot model and validate its performance on a small subset of validation instances to ensure adequate training while avoiding overfitting. (2) A subset of child models is randomly selected and evaluated on the same validation subset using the inherited weights from the one-shot model. (3) The child models with the best performance will be derived and retrained for the final recommendation outcome. Moreover, to reduce the magnitude of the search space and find the better recommendation models, AMEIR modularizes the search process into three steps matching the three stages in the backbone model by progressively searching for the architectures in the three search subspaces [Li et al., 2019;Sciuto et al., 2020]. Meanwhile, AMEIR ensures the fairness [Chu et al., 2019] of one-shot model training to alleviate the representation shift caused by weight sharing [Chu et al., 2019;Yu et al., 2020b], where all candidates are uniformly sampled and activated per gradient update. Details of the three steps are shown below.
Step 1: Behavior Modeling Search. In Step 1, AMEIR searches for the behavior modeling networks from the search subspace of building blocks, as shown in Fig.1. The normalization, layer, and activation operations are randomly selected from the corresponding operation sets to construct the recommendation model. To decouple behavior modeling from other stages, a pre-defined MLP is exploited to combine h a and e b during Step 1. The training and evaluation procedures are the same as previously described.
Step 2: Interaction Exploration. In Step 2, AMEIR combines one-shot NAS with the sequential model-based optimization (SMBO) algorithm [Liu et al., 2018a] to explore interactions on non-sequential features. The interaction set p b is initialized with non-sequential features e b , and then progressively updated by several cycles of evolution to increase the interaction order, as illustrated in Fig In order to train and evaluate interactions in the one-shot model, AMEIR utilizes another pre-defined MLP to aggregate interactions in p b with non-sequential embeddings in e b , while the behavior modeling here is left out for efficiency. As the linear projection of feature concatenation is equivalent to feature-wise addition of projected results, i.e., Step 2 is efficiently realized by adding the projections of one uniformly-sampled interaction and all embeddings in e b as the output of the first hidden layer, without accessing to all interactions in p b . To further speed up the search process, the weights of the one-shot model are inherited in consecutive evolution cycles.
Step 3: MLP Investigation. In Step 3, AMEIR investigates the dimensions and activation functions in the aggregation MLP based on the behavior modeling network and interactions found in the previous steps. To find the hidden sizes of MLP, AMEIR pre-allocates a weight matrix with the maximum dimension of (K c 0 , K c 0 ), i.e., W c ∈ R K0×K0 , for each hidden layer [Guo et al., 2019]. Suppose h in and h out are chosen to be the hidden sizes of previous and current layers. AMEIR will slice out sub-matrix W c [: h in , : h out ] to assemble the MLP in Eqn. (4). Other training and evaluation procedures are similar to Steps 1 and 2. Derivation. After the three-step random search, AMEIR assembles the search results as the final outcome and retrains the derived model for evaluation. It is worth emphasizing that the derived model is initialized with the one-shot model weights because the training protocols of model architectures and datasets are the same for both one-shot model and child models, which is in contrast to the most existing methods that adopt proxy task for one-shot model training So et al., 2019;Xie et al., 2019;Yu et al., 2020b]. The inherited weights accelerate the retraining process and compensate for the search cost.

Related Works to AMEIR
Behavior Modeling.
The existing methods introduce RNN [Hidasi and Karatzoglou, 2018], CNN [Yuan et al., 2019], or Transformer-based [Sun et al., 2019] behavior modeling networks to capture sequential relations and evolved interests from user behaviors. Then, the sequential features are extracted from the sequence-level hidden states by either taking the last hidden state [Kang and McAuley, 2018] or aggregating the sequence by a pooling layer [Cheng et al., 2016;Covington et al., 2016]. Some recent works [Zhou et al., 2018b;Zhou et al., 2019] further employ an attention mechanism between the sequential representations and target item to retrieve the user's main proposal. These methods can all be found in AMEIR's search space. Specifically, the search subspace of building blocks contains all layer, activation, and normalization operations in those behavior modeling networks. Moreover, the last hidden state, pooling-based aggregation, and attention-based aggregation are all used for sequence compression. Interaction Exploration. In parallel with behavior modeling, some works resort to explore feature interactions in the DL-based recommendation models. LR  [Lian et al., 2018] are exploited to import explicit low-order or high-order interactions through shortcut connections, i.e., concatenating interactions to the last hidden layer of MLP, and then apply linear projection with learnable variables or all-one vectors to generate the prediction scores jointly with the hidden layers. Other than the shortcut connection, FibiNet [Huang et al., 2019] take interactions as the input of MLP, while PNN [Qu et al., 2016;Qu et al., 2018] further combines interactions with raw embeddings to induce more features in the network. It is not hard to find that all these methods can be derived from AMEIR's search space. Concretely, all low-order and highorder interactions can be extracted from the search subspace of interactions. Meanwhile, raw input embeddings and explored interactions are used by both shortcut connection and MLP, covering the existing interaction-combining strategies. Besides, the attentive SE connection could implicitly characterize attention-based, regression-based, and FM-based shortcut connections. MLP Investigation. The existing literature [Covington et al., 2016;Guo et al., 2017;Zhou et al., 2018b] mainly uses suboptimal hand-crafted MLPs to aggregate features, where activation functions and hidden sizes are designed manually or by grid search. However, the search space of MLP is large and these hand-crafted MLP settings are always sub-optimal. AMEIR introduces a discrete yet flexible search space that covers most of the common MLP settings and finds the MLP  Based on the above analysis, it can be inferred that AMEIR's search space covers most of the representative recommendation models. Therefore, AMEIR can perform as good as these methods in various scenarios, and further derive better models when conducting sufficient training and exhaustive search. A more detailed comparison between AMEIR and the related works is given in Appendix A.

Experiments
In Sec.3, we will compare AMEIR with competitive baselines on various recommendation scenarios, study the impact of different components, and conduct the efficiency analysis on the proposed method. We will also describe more details of the experiments in Appendix B-E, including datasets, baseline methods, implementations, and search results.

Industrial Results for AMEIR
To verify all three steps of AMEIR in the industrial scenario, we refer to the Alimama CTR dataset [Feng et al., 2019] that comprises both sequential and non-sequential features for a comprehensive comparison. We use similar preprocessing settings as [Feng et al., 2019], where the embedding size is set to 4 and the maximum sequence length is set to 50. The Adam optimizer with an initial learning rate 1e-5 and batch size 1024 is employed for both AMEIR and baseline methods. For the training of the one-shot model, a single cosine schedule is introduced for learning rate decay. In Steps 1 and 3, AMEIR selects the top-5 models from the randomly sampled 2000 architectures for further evaluation, while in Step 2, AMEIR conducts 4 cycles of evolution, retains top-50 interactions at each cycle, and finally reserves M = 5 interactions as p b . L a and L c are set to 3 and 2 in the backbone model, and the pre-defined MLP is set to [200,80] during Steps 1 and 2. By conforming to [Li and Talwalkar, 2019;, we run AMEIR for 4 times and report the mean metrics across 4 runs as the final results.
Tab. 1 depicts the results on Alimama dataset. AMEIR-A (only uses behavior modeling result from Step 1) presents competitive performance with the promising empirical architectures DIEN and DSIN. Furthermore, a significant improvement (note that an improvement of AUC around 0.0005-0.001 is already regarded as practically significant in the industrial scenarios [Feng et al., 2019]) can be observed when combining AMEIR-A with searched interactions (AMEIR-B) and MLP (AMEIR-C). An interesting point on the Alimama dataset is that compared to sequential modeling and interaction exploration, the MLP search significantly contributes to the final results, which indicates that all three stages/steps in AMEIR are necessary for high-quality recommendations.

Applicable for Various Scenarios
To show AMEIR is applicable for various recommendation tasks, we evaluate AMEIR on two additional scenarios, namely sequential scenario, and non-sequential scenario: (1) The sequential scenario only presents sequential user behaviors to examine Step 1 behavior modeling. The training is based on the item retrieval task. The sequential datasets include Amazon Beauty and Steam [Kang and McAuley, 2018].
(2) The non-sequential scenario is introduced to verify the Step 2

Ablation Study
Ablation Study on NAS Methods. To study the importance of the proposed search method, we evaluate the performance of different NAS algorithms on the same search space of the Alimama dataset. As shown in Tab. 4, the one-shot random search achieves better performance than both brute-force methods of random, RL, EA, and the algorithmic complex methods of ENAS, DARTS, SNAS with lower search cost. Moreover, AMEIR with the three-stage one-shot searching pipeline brings about a remarkable improvement on the oneshot random search under a similar time budget, demonstrating the effectiveness and efficiency of the proposed method. Ablation Study on Backbone Model. To further analyze the impact of the backbone model, we isolate different components in Stages 2 and 3, and report the mean metrics on the non-sequential datasets. As shown in Tab. 5, the interactions (searched) applied to MLP and FM bring orthogonal improvement on vanilla DNN. The result reveals that both combining strategies contribute to the result. Additionally, SE interaction (AMEIR-B) and MLP search (AMEIR-C) could further boost the performance of recommendation, suggesting the superiority of the backbone design in AMEIR.

Efficiency in Search and Inference Phases
Search Efficiency. The search cost of AMEIR on the Beauty, Steam, Avazu, Criteo, Alimama datasets are 0.2, 0.2, 0.25, 0.25, 0.5 GPU-days respectively on a single Tesla V100 GPU, which are all similar to training the individual models of baseline methods. As AMEIR uses the trained one-shot weights to initialize the derived model, the fine-tuning time could generally be ignored. Therefore, the overall time cost of AMEIR is roughly the same as the methods that do not employ NAS. The result proves the efficiency of AMEIR in the search phase, and it is evident that AMEIR can help facilitate the design of recommender systems in real-world scenarios. Inference Efficiency. We record the queries/samples-persecond (QPS) of AMEIR and the compared models to examine the inference speed on the Alimama dataset in Tab. 4. Apart from the improvement in accuracy, AMEIR is also competitive in efficiency compared with the commonly-used attention-based models. Besides, the searched MLPs are always much smaller than the hand-crafted ones (details shown in Appendix E), such that the numbers of parameters of the searched models are less than half of the baselines. The results indicate that instead of increasing the model complexity, AMEIR coordinates the three parts of a recommender system to find the architecture with the best practice.

Conclusion
In this paper, we introduce AMEIR for automatic behavior modeling, interaction exploration, and MLP investigation in the recommender systems to relieve the manual efforts of feature engineering and architecture engineering. Owing to the three-stage search space and the matched three-step searching pipeline, AMEIR covers most of the representative recommendation models and outperforms the competitive baselines in various recommendation scenarios with lower model complexity and comparable time cost, demonstrating efficiency, effectiveness, and robustness of the proposed method.

A.1 Behavior Modeling
The existing behavior modeling methods can be categorized into RNN-based, CNN-based, Transformer-based methods, and mixed methods.
• RNN-based methods [Hidasi et al., 2016;Hidasi and Karatzoglou, 2018;Quadrana et al., 2017; mainly apply LSTM or GRU layers to extract sequential representations, which can be covered by the building blocks with recurrent layer operations.
• CNN-based methods apply convolutional layers on the sequential representations. Caser [Tang and Wang, 2018] introduces horizontal and vertical convolutions by sliding windows on the sequential embedding and applies max pooling over the extracted feature maps for compression. However, sequence-level pooling on one hidden convolution layer is hard to capture complex relations in the sequence. NextItNet [Yuan et al., 2019] utilizes stacked residual blocks of dilated convolution, which can be roughly regarded as a special case of the behavior modeling network in AMEIR because both dilated convolutional block and standard 1 × 1 convolutional block are included in the search subspace of Stage 1.
• Transformer-based methods, e.g., ATRank [Zhou et al., 2018a] On top of the behavior modeling, these methods either extract the last hidden state [Hidasi et al., 2016;Kang and McAuley, 2018], or employ pooling layer [Tang and Wang, 2018] and attention layer from target [Zhou et al., 2018b;Zhou et al., 2019] to compress the sequential hidden states into a fixed-length representation, which are all comprised by the behavior modeling module of Stage 1 in AMEIR.

A.2 Interaction Exploration
In Sec.2, we have declared that AMEIR can cover all feature interactions and interaction-combining strategies in the representative interaction exploration methods. For a more formal and intuitive comparison, we develop the computation process of final prediction scores in both AMEIR and the representative recommender systems and show that all compared methods can be explicitly or implicitly included in AMEIR's search space. Following the existing symbol usage, x b and e b denote the non-sequential input features and corresponding field embeddings. For the convenience of representation, we use p b to represent both investigated interactions and enumerated bounded-degree interactions of various types, including cross-product interactions, bilinear interactions, and Hadamard-product interactions (inner-product interaction can be expressed as a special form of Hadamard-product interaction, i.e., 1 T p b i , where p b i represents a Hadamardproduct interaction) according to the related literature. We use a to denote the attention weights of interactions and embeddings computed by compressed interaction network (CIN) [Lian et al., 2018] or attention module [Liu et al., 2020;Xiao et al., 2017]. The bias in the MLP layers is omitted for convenience. We compare AMEIR with representative methods of LR, FM [Rendle, 2010] where w . Therefore, all compared methods are explicitly or implicitly contained in AMEIR's search space.

B Datasets
In Appendix B, we introduce the datasets in the industrial scenario, sequential scenario, and non-sequential scenario. Tables 6-8 show the statistics of the datasets. The Alimama dataset in the industrial scenario contains data of all formats mentioned in Sec. 3 so that it is used for comprehensive comparison in all three-stage pipeline of AMEIR: Alimama 1 provides 26 million records of ad display/click logs in 8 days, where for each user it keeps track of 200 recent shopping behaviors in 22 days. The logs in the first 7 days are used for training and the logs in the last day are treated as the test set. Each data instance has 15 sparse features, including user profile, user's sequential behaviors, and context profile. The dimension of these features ranges from 3 to 309,449. There is only 1 dense feature of "price" which has been normalized to [-1,1]. We randomly sample 25% of the dataset and remains 6,204,907 data instances following [Feng et al., 2019]. 5% of the training set is used for validation in the architecture search and model selection. As there are many users that do not have any records of sequential behaviors, we assign a special tag to the behavior sequences of these users to avoid the behavior modeling network executing on vacant inputs. The Amazon Beauty and Steam datasets in the sequential scenario are proposed to evaluate the Stage 1 behavior modeling in AMEIR, which only include sequential behavior of each user, and the training is based on next-item prediction with Eqn. (7):

B.2 Sequential Scenario
• Amazon 2 is composed of reviews and metadata from Amazon.com [McAuley et al., 2015], where data instances are categorized into separate datasets according to the product category. In this paper, we use the category of "Beauty" for evaluation. , 2018] which contains millions of review information from the Steam video game platform.

• Steam 3 is introduced by [Kang and McAuley
Following the common practice [He et al., 2017b;Kang and McAuley, 2018], the reviews in the datasets are treated as implicit feedback. We group data instances by users and sort them according to the timestamps. The users and items with less than 5 feedbacks are removed from datasets. The leave-one-out strategy is adopted for partitioning [He et al., 2017b;Kang and McAuley, 2018;Sun et al., 2019], where for each user the last two actions are treated as the validation set and test set, and the remaining actions are used for training. The Criteo and Avazu datasets in the non-sequential scenario only include non-sequential categorical features for CTR prediction to verify the Stage 2 interaction exploration and Stage 3 aggregation MLP investigation:

B.3 Non-sequential Scenario
• Criteo 4 contains 7 days of click-through data, which is widely used for CTR prediction benchmarking. There are 26 anonymous categorical fields and 13 continuous fields in the Criteo dataset. The continuous fields are normalized to [-1,1] and only used for MLP input while categorical fields are applied to both MLP and shortcut connection as well as the interaction exploration module in AMEIR. The dataset is randomly split into two parts while maintaining the label distribution: 90% is for training and the rest is for testing. Besides, 10% of the training set is used for validation.
• Avazu 5 consists of 11 days of online ad click-through data. Each data instance includes 24 categorical fields that represent profiles of a single ad impression. We randomly split the dataset into two parts, where 80% is for training and the rest is for testing. Same as the Criteo dataset, 10% of the training set is used for validation.

C Baseline Methods
In Appendix C, we introduce baseline methods that are adopted for comparison in the industrial scenario, sequential scenario, non-sequential scenario, and ablation study on searching methods.

C.1 Industrial Scenario
The • YouTube DNN [Covington et al., 2016] is usually introduced as the base model in CTR prediction. It utilizes sum pooling on the users' historical behavior embedding and then concatenates the results with other non-sequential embeddings of the user profile, context profile, and target item/ad profile as the input of MLP to generate the final prediction score.
• Wide & Deep [Cheng et al., 2016] additionally adds a shortcut connection of linear regression on the non-sequential features to the output of MLP to capture both generalization and memorization.
• DeepFM [Guo et al., 2017] replaces the linear regression-based shortcut connection in Wide & Deep with a Factorization Machine to involve feature interactions in the recommendation model.
• DIN [Zhou et al., 2018b] extends the base MLP model by using an attention mechanism on the user behavior with respect to the target item to adaptively learn the diverse characteristic of user interests.
• DIEN [Zhou et al., 2019] improves DIN by using GRU and AUGRU module to the extract user's evolving interests, where an auxiliary loss is employed to make hidden states more expressive.
• DSIN [Feng et al., 2019] further investigates the homogeneity and heterogeneity in the user behavior and models the user interests through intra-session self-attention layer and inter-session Bi-LSTM layer. The sequential user behaviors are split into sessions by 30-minute intervals, and each session remains at most 10 behaviors.
The aggregation MLP of all baseline methods is fixed to [200,80], while the activation function is set to Dice in DIN, DIEN, and set to ReLU in other methods. The training is based on the CTR prediction of Eqn. (6) for all methods.

C.2 Sequential Scenario
To evaluate AMEIR in the sequential scenario, we introduce several baselines for comparison. We carefully categorize the stateof-the-art sequential recommender systems and select the representative methods for each category to report the performance in the limited pages of the paper. Since other sequential recommendation methods have been outperformed on the similar datasets by the selected baselines, we omit comparison against them.
• POP is the simplest baseline which only recommends the most popular items with the highest number of interactions from the candidate set (also used for evaluating the complexity of datasets). • NextItNet [Yuan et al., 2019] employs residual blocks of dilated convolutions to increase receptive fields and model both long-range and short-range item dependencies on user behaviors. We set the kernel size of convolutions as 3, and leverage 5 dilated convolutional layers with dilation 1, 2, 1, 2, 2.
• SRGNN [Wu et al., 2019b] applies GNN to generate item representations and then encodes global and current user interests using an attention network. We use the Gated-GNN to learn the item embeddings, add regularization of all parameters with weight 1e-6, and set the initial learning rate to 1e-7 to avoid the gradient explosion of the recurrent unit. SRGNN is the classical implementation to use GNN on recommendation, which represents the GNN-based recommender system such as GCSAN [Xu et al., 2019], GraphRec [Fan et al., 2019] and PinSage .
• BERT4Rec [Sun et al., 2019] adopts BERT [Devlin et al., 2019] to learn bi-directional representations of sequences by predicting masked items in the context. The transformer is set to have 2 layers with 4 heads. The dropout rate is set to 0.5 on inputs, 0.2 on feed-forward networks after each multi-head attention and convolution to prevent overfitting. BERT4REC is a successful transplant of Transformer to recommendation, which is on behalf of Transformer-based recommender systems including SASREC [Kang and McAuley, 2018], MBTREC [Zhang et al., 2020], BST [Chen et al., 2019a], ATRank [Zhou et al., 2018a], etc.
The model-relevant hyper-parameters and initializations are either following the recommended settings in the corresponding papers or tuned by the grid search on the validation sets. We employ the same training process as AMEIR for these methods.

C.3 Non-sequential Scenario
The baseline methods in the non-sequential scenario include representative interaction exploration methods in deep learningbased recommender systems. YouTube DNN [Covington et al., 2016] and Wide & Deep [Cheng et al., 2016]  • IPNN [Qu et al., 2016] combines the inner product of feature interactions with raw feature embeddings as MLP input.
• DCN  proposes cross network to comprise explicit bit-wise interactions of bounded degrees. The compressed interactions and the output of MLP are concatenated and then fed to a linear layer to generate the prediction scores.
• xDeepFM [Lian et al., 2018] designs compressed interaction network (CIN) to replace cross network in DCN  by enumerating vector-wise interactions via Hadamard product. The linear regression is also applied for shortcut connection as DeepFM. • AutoCTR [Song et al., 2020] modularizes representative operations of one-layer MLP, dot product (DP) and factorization machine (FM) to formulate the direct acyclic graphs search space for the recommendation model. It performs an evolutionary algorithm with learning-to-rank guidance during architecture search. The data sub-sampling and hash size reduction are used for acceleration.
To ensure fair comparison, we use the same pre-defined aggregation MLP in AMEIR for the baseline methods, i.e., [200 × 3] for Criteo [Guo et al., 2017] and [500 × 5] for Avazu [Qu et al., 2018] with ReLU activation. Besides, CIN is set to 256 / 128 layers for Criteo / Avazu in xDeepFM. The training is based on the CTR prediction of Eqn. (6) for all methods.

C.4 Ablation Study on Search Methods
The baseline search methods include the brute-force random search, reinforcement learning (RL), evolutionary algorithm (EA), and the popular one-shot methods ENAS, DARTS, SNAS. Conforming to the backbone settings in Sec.3, we select the best performing behavior modeling network, top-5 feature interactions, and top-1 aggregation MLP in different stages for all methods.
• Random [Bergstra and Bengio, 2012] is the baseline NAS method that simply samples architectures from a uniform distribution over the search space. For a fair comparison, we randomly select 300 child models from the search space until the total training time plus the evaluation time reaches 10 times the search cost of the proposed one-shot random search. All child models are trained by 1 epoch according to [Dong and Yang, 2019a;Radosavovic et al., 2020], and the top-5 architectures with the highest validation accuracy will be selected for further evaluation to pick the best one as the final result. • RL [Zoph and Le, 2017;Zoph et al., 2018] is another baseline method of NAS. At each round, a controller generates a child model from the search space. The child model is then trained and evaluated on the validation set, and the result is treated as the reward to train the weights of the controller for a better outcome. In our implementation, we use a two-layer LSTM as the controller with the hidden size of 32. The controller is trained by REINFORCE rule [Williams, 1992] with Proximal Policy Optimization [Schulman et al., 2017], with a 0.9 exponential moving average of the previous rewards as the baseline [Pham et al., 2018]. We also use the entropy penalty, temperature 5, and the tanh constant 2.5 for the sampling logits [Pham et al., 2018]. Each child model is trained for 1 epoch that is the same as the random search. The searching process is finished once the performance of the consecutively generated models does not grow or reaches the 10-time budgets as the one-shot random search. Similar to the Random, top-5 models in history will be selected for more evaluation. • EA [Real et al., 2017;Real et al., 2019] maintains a population of child models, and gradually evolves the population for better results. In this paper, we re-implement the regularized evolutionary algorithm [Real et al., 2019] for comparison. Specifically, we first uniformly sample several child models from the search space as the initial population. To mutate the population, we sample one random child model as the parent from the population at each cycle, and then construct a new model from the parent by a mutation, which causes a simple modification on the network architecture, e.g, altering a layer operation in Stage 1, a feature interaction in Stage 2, or an MLP hidden size in Stage 3. Once the child architecture is constructed, it is then trained, evaluated on the validation set, and added to the population. Meanwhile, the oldest member of the population will be removed to maintain the population size. We set the initial population size as 10, and evolve only one architecture at each cycle. Each child model in the population is trained by 1 epoch. Once the population does not change for over 5 cycles or reach the 10-time budgets as the one-shot random search, the algorithm will be finished, and the top-performing models will be derived from the population. • ENAS [Pham et al., 2018] introduces the one-shot strategy in the RL algorithm. The only difference between ENAS and RL is that the sampled child models in ENAS share the same weights in the one-shot model. For the fair comparison, we use the same training and evaluation settings in all the search methods.

D Implementation Details
In Appendix D, we introduce the implementation details of AMEIR and baseline methods, including the common settings and the selection of hyper-parameters in industrial, sequential, and non-sequential scenarios.

D.2 Sequential Scenario
For the sequential scenario, we fix the maximum sequence length as 15 for Beauty and 25 for Steam. The initialization is the same as industrial scenario, and the models are optimized by Adam optimization with batch size 512. The input sequences are randomly clipped for the model training, with the last five items served as the positive items and another five randomly sampled items that the user has not interacted with served as the negative items. We also apply grid-search for the hyper-parameter tuning in the sequential scenario. We search the embedding size from {64, 96, 128, 160, 192} and find that when the embedding size is greater than 128, the performance of most of the baselines would not further increase and the compared models even become hard to reach convergence. Therefore, we fix the embedding size as 128 for all models. We also search for the initial learning rate from {1e-3, 1e-4, 1e-5}, L2 regularizer from {1e-3, 1e-4, 1e-5, 1e-6}, and dropout ratio of building blocks from {0, 0.1, 0.2, 0.3, 0.4, 0.5}.

E Search Results
In Appendix E, we show the searched building blocks, feature interactions, and aggregation MLPs of AMEIR and the corresponding experimental results in the 4 runs.

E.1 Industrial Scenario
Tab. 9 demonstrates the searched architectures on the industrial dataset. Regarding step 1, AMEIR automatically raises the best behavior modeling network with two or three building blocks based on the sequential data. It can be observed that the searched blocks over the 4 runs all include one non-parameterized operation of average pooling or dummy layer of zero, indicating that the shallow sequential models might be optimal for this dataset. Therefore, compared to the empirical stateof-the-art CTR prediction models such as DIEN and DSIN, the searched building blocks contribute more to the AUC metric.
The search results also demonstrate that the one-shot training is capable of finding architectures of appropriate depth to prevent pre-mature convergence of the shallow models and avoid overfitting of the deep models. Besides, although the "attention from target" operation is involved in the industrial scenario, we find that it does not help improve the AUC result. Regarding step 2, the searched interactions contain both second-order and third-order interactions. Though there are few non-sequential sparse features in the Alimama dataset, the searched interactions can slightly but consistently improve the performance upon the searched blocks. Note that some interactions, e.g., (F04, F10), (F05, F06, F11), and (F06, F11) are shared across different runs. The phenomenon verifies the robustness of AMEIR and enhances the interpretability of the prediction. Regarding step 3, the searched aggregation MLP tends to select the layers with lower dimension activated by ReLU. This decision significantly upgrades the performance on the CTR prediction task, with fewer parameters compared to the models by manual architecture engineering. The prominent performance of the low-dimension MLP enlightens us that a proper MLP could boost the learning capability of a specific architecture, and one-shot searching is able to acquire the best settings of the MLP. Additionally, compared to the novel activations of Swish and Dice, AMEIR prefers ReLU in the searched MLP, which demonstrates that using all ReLUs as the activation functions might be a good starting point for MLP design. The above observations verify the practicability and efficiency of both the three-stage search space and the three-step one-shot searching pipeline of AMEIR.

E.2 Sequential Scenario
Tab. 10 summarizes the searched building blocks in the sequential scenario with the corresponding experimental results. Similar to the industrial scenario, though the maximal number of layers is set to 6, all searched architectures for each dataset contains three to five layers (i.e., at least one "Zero" in each searched architecture), indicating that the shallow models might be more suitable for Beauty and Steam datasets. It is also evident that the one-shot search tends to raise the self-attention block and convolutional block in those two datasets and construct similar architectures to the state-of-the-art manual designs (e.g., #1 of Beauty to NextItNet, #1 of Steam to BERT4REC). Beyond that, the layer normalization is usually deployed on the shallow layers, and the Bi-GRU layer mostly locates at the deep layer. We believe that these placements contribute to the efficient knowledge extraction from user behaviors as well as the fast convergence of sequential modeling.

E.3 Non-sequential Scenario
Tab. 11 depicts the searched interactions, as well as the pre-defined / searched MLP of AMEIR-B and AMEIR-C in the nonsequential scenario. Though AMEIR could search high-order interactions, most of the searched interactions (best-performing interactions) are in second-order while some pairs of features never form appropriate interactions to be selected by the oneshot model according to the statistics. This observation attributes to the automatic recognition of the feasible interactions with appropriate order by the one-shot searching, potentially relieving the effort of manual feature engineering. Same as the industrial scenario, the shared interactions can also be found in the non-sequential scenario, e.g., (C6, C13), (C13, C26) in Criteo and (C15, C18), (device conn type, C18) in Avazu, once again verifies the robustness of AMEIR over multiple runs.
Regarding the aggregation MLP, we notice that AMEIR-C tends to select MLPs of the higher dimension in the bottom layers and lower dimension in the top layers, which is more reasonable for the neural network design as opposed to the hand-crafted settings in [Guo et al., 2017;Qu et al., 2018]. The adaptive design of MLP further improves the performance on the nonsequential datasets, indicating the effective and efficient one-shot searching of AMEIR. Besides, the preference of ReLU also appears in the non-sequential scenario.

F Discussion on One-shot Strategy
The one-shot strategy has been widely used for accelerating the neural architecture search. In most of the existing literature Xie et al., 2019;Wu et al., 2019a;Guo et al., 2019], the one-shot NAS is only applied to derive the best model from the search space, and the derived model is re-trained from scratch. In AMEIR, we find an interesting phenomenon that fine-tuning the derived model with the one-shot model weights can achieve better performance than directly re-training the derived model with random initialization. The phenomenon is aligned with the recently proposed Once-for-All [Cai et al., 2019a], BigNAS [Yu et al., 2020a] and AttentiveNAS [Wang et al., 2020a], which only trains the network once and directly use the one-shot weights for evaluation without re-training. These methods achieve similar or even better performance than training the child models from scratch, which validates our experimental results. Moreover, we observe a larger improvement in using one-shot weights as the initialization in AMEIR. We guess that this is because the recommendation models usually encounter the overfitting problem, and the one-shot training, which is equivalent to a adding large drop-path strategy on the one-shot model, can be regarded as introducing an extra regularization on the model training. Therefore, it helps prevent/delay the appearance of overfitting meanwhile guaranteeing sufficient training of the network. We observe that under similar training settings, one-shot training presents a later overfitting epoch compared to re-training the derived models from scratch, which is consistent with our argument.  Table 11: Searched interactions, MLPs and corresponding experimental results of interaction exploration (AMEIR-B) and additional aggregation MLP investigation (AMEIR-C) on Criteo and Avazu over 4 runs. Each interaction is represented by a tuple of non-sequential feature fields. Each MLP includes a list of hidden layers, where each hidden layer is represented by a tuple of (hidden size, activation operation).