{'fraction_non_alphanumeric': 0.03486251914521187, 'fraction_numerical': 0.006296647460676342, 'mean_word_length': 5.283837457989612, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multiplex Behavioral Relation Learning for Recommendation via Memory Augmented Transformer Network

Capturing users' precise preferences is of great importance in various recommender systems (eg., e-commerce platforms), which is the basis of how to present personalized interesting product lists to individual users. In spite of significant progress has been made to consider relations between users and items, most of the existing recommendation techniques solely focus on singular type of user-item interactions. However, user-item interactive behavior is often exhibited with multi-type (e.g., page view, add-to-favorite and purchase) and inter-dependent in nature. The overlook of multiplex behavior relations can hardly recognize the multi-modal contextual signals across different types of interactions, which limit the feasibility of current recommendation methods. To tackle the above challenge, this work proposes a Memory-Augmented Transformer Networks (MATN), to enable the recommendation with multiplex behavioral relational information, and joint modeling of type-specific behavioral context and type-wise behavior inter-dependencies, in a fully automatic manner. In our MATN framework, we first develop a transformer-based multi-behavior relation encoder, to make the learned interaction representations be reflective of the cross-type behavior relations. Furthermore, a memory attention network is proposed to supercharge MATN capturing the contextual signals of different types of behavior into the category-specific latent embedding space. Finally, a cross-behavior aggregation component is introduced to promote the comprehensive collaboration across type-aware interaction behavior representations, and discriminate their inherent contributions in assisting recommendations. Extensive experiments on two benchmark datasets and a real-world e-commence user behavior data demonstrate significant improvements obtained by MATN over baselines. Codes are available at: https://github.com/akaxlh/MATN.


INTRODUCTION
Recommender system, which facilitates the information-seeking process of users and meet their personalized interests, have played a critical role in various online services, such as e-commerce systems [13,38], online review platforms [1,44] and advertising [39]. At its core is to learn low-dimensional representations of user-item interaction while capturing the user preference and the underlying intrinsic characteristics [19]. Early methods towards this goal, have made significant efforts on transforming user-item interactions through vectorized representations based on the conventional Collaborative Filtering (CF) techniques (e.g., matrix factorization scheme [15,22] and its variations [12,26]).
Inspired by the advancement of deep learning techniques, various neural network-based collaborative filtering frameworks have been developed to model the relationships between users and items. These methods aim to map sparse input interaction features into low-dimensional user/item embedding vectors and then project them into fixed-length representations in a group-wise manner [16,54]. For example, neural collaborative filtering models replace the inner product function in the matrix factorization consider non-linearities with multilayer perceptron [11] and metric learning scheme [33]. In addition, auto-encoder architecture has served as an effective solution to learn a mapping function between the explicit interaction and latent representation through the reconstructionbased encoder-decoder framework. To capture the rich graph-based neighborhood contextual signals, various graph neural encoders have been proposed to aggregate information over the user-item interaction graph, with the graph convolutional network [49] or message-passing mechanism [41].
Despite the prevalence of the above recommendation solutions, they has thus far focused on user preference representation learning with the consideration of singular type of user-item interactive behavior. However, in many practical recommendation scenarios, user-item interactions are multiplex and exhibited with relationship diversity in nature. Let's consider the e-commerce system as an example, there exist multiple types of behavior (e.g., page view, add-to-favourite, add-to-cart and purchase) between users and items [8], which are mutually inter-dependent. For instance, add-to-cart behavior is more likely to co-occur with purchase than the add-to-favorite behavior. The page view and add-to-favourite behavior can also provide useful signals for making purchase decisions. In such cases, the ignorance of such multi-modal relations across different types of user-item interaction behavior, makes existing recommendation methods insufficient to distill effective collaborative signals from the collective users behavior.
The recommendation framework with multiplex interactive behavior pose two key challenges: First, the dependencies across different types of user-item interactions can be arbitrary since any pair of type-specific behavior could potentially be correlated due to various factors [42]. For example, users often have correlated online behaviors and exhibit different dependencies in choosing items of different categories due to his/her specialty. Such interdependencies between different types of interaction behavior may vary by users and items. While a handful of studies attempt to learn user preference from multi-behavior [7,8], they merely consider the singular dimensional cascading correlations between multi-type interactions, and cannot comprehensively capture the arbitrary dependencies between different types of interaction over different items. Hence, to build effective recommendation model with the complex behavior dependencies remains a significant challenge.
Second, when modeling the relationships across different types of behavior, it is also important to capture the context and semantics of individual type of user-item interactions, e.g., users' page view are more frequent than their purchases, and add-to-favorite behavior is more likely to happen over users' interested items but may postpone their buying decision. In addition, type-specific behavioral patterns interweave with each other in complex way (e.g., support or mutually exclusive relations) and are difficult to be captured. During the behavioral pattern integration, as the importance of various types of behavior can be different, their relevance in assisting the forecasting task on the target behavior need to be carefully decided.
Motivated by the aforementioned challenges, this work proposes a general and flexible multi-behavior relation learning framework-Memory-Augmented Transformer Networks (MATN). Specifically, this work first proposes a multi-behavior transformer network to learn type-specific behavioral representations with the incorporation of inter-dependencies across different types of user-item interactions. By integrating the transformer network with a memoryaugmented attention mechanism, we endow the MATN framework with the capability of incorporating type-specific behavior contextual signals. to collectively model the implicit relevance across multi-type behavioral patterns and perform comprehensive learning for making recommendations, we design a behavior type-wise gating mechanism which promotes the collaboration of different types of interactions. In the pattern aggregation layer, MATN could learn cross-type representations in the latent feature spaces by automatically adjusting the contribution of each behavior view point in the behavior predictive model.
The contributions of this paper are highlighted as follows: • We propose MATN, a new recommendation framework with multiplex behavioral relation learning. MATN explicitly encodes multi-behavior relational structures by preserving both the crosstype behavior collaborative signals and type-specific behavior contextual information.
• We first develop a multi-behavior dependency encoder with a transformer architecture, to inject collaborative signals across different types of user-item interactions into the embedding process. Furthermore, we augment the multi-behavior transformer network with a memory attention mechanism, which is capable of uncovering type-specific behavior semantics during the customized representation recalibration phase.
• Finally, a type-wise pattern aggregation layer with gating mechanism is developed to promote the collaboration of different behavior views for robust representations on user preferences.
• Our extensive experiments on two benchmark datasets and a user behavior data from a major e-commence platform, demonstrate that MATN outperforms 12 baselines from various research lines in yielding better recommendation performance. We further perform case studies with qualitative examples to better understand the interpretation ability of MATN framework, and study the model efficiency under different recommendation scenarios.

PRELIMINARY
In the recommendation scenario, we first define the behavior (e.g., purchase) which we aim to predict as target behavior, other relevant user-item interactive behavior (e.g., click, add-to-cart and add-to-favorite) is termed as source behavior. In this work, we aim to explore the latent relational structures between different types of user behavior (e.g., purchases and click) for making predictions on the target behavior of users in recommender systems. Definition 1. Multi-Behavior Tensor X. We define a threedimensional multi-behavior tensor X ∈ R × × to represent the (indexed by ) types of behavior from (indexed by ) users over (indexed by ) items. Without loss of generality, we focus on the implicit user feedback which is more common in practical recommendation scenarios [25,40]. Particularly, in tensor X, each entry , , = 1 if user is interacted with item given the -th behavior type. For example, if user purchases item , the corresponding element , , will be set as 1 in the purchase behavior matrix X .
Problem Statement. Based on the aforementioned definitions, the recommendation task with multiplex behavior learning is formulated as follows: Input: the user-item interaction data represented with multi-behavior tensor X (including both the target and source behavior). Output: A predictive model to effectively infer the unknown user-item interactions in X with the target behavior .

METHODOLOGY
In this section, we present the technical details of MATN framework, the architecture of which is illustrated in Figure 1. MATN is a hierarchical neural architecture with three key modules in MATN: (i) cross-behavior embedding layers that learn the representations by exploring the inter-dependencies across different types of interactions; (ii) a customized representation recalibration network that refines the latent embeddings, with the preservation of individual behavioral contextual information; (iii) a forecasting layer that aggregates the refined behavior type-specific embeddings and outputs a predicted likelihood of a user-item interaction pair.

Multi-Behavior Dependency Modeling
As discussed before, different types of user behaviors are correlated with each other, which brings in new challenges to the recommendation framework. To model the inter-dependencies across different types of behavior, we design a multi-behavior transformer network to promote the collaboration of different behavioral views. To achieve this goal, we learns a robust representation for useritem interactive patterns of each individual categorical behavior , which integrates the relevant information from other behavior views ′ ∈ [1, ..., ]& ′ ≠ .
3.1.1 Initialized Embedding Layer. Firstly, a projection layer is introduced to map the original multi-behavior user-item interaction data into initial latent representations. We denote the interaction vector of -th behavior type and -th user ( ) over all items ( , 1 ≤ ≤ ) as X , ∈ R . The projection operation for X , is formally defined asX , = V · X , , where V ∈ R × and denotes learned projection matrix and hidden state dimensionality, respectively. Note that V is shared across behavior categories to model the common semantics of different interactions. The projectedX , serves as an initial parameterized state for user-item interactions X , , to be optimized with the following modules.

3.1.2
Multi-Head Self-Attentive Mechanism. Inspired by the promising potential of self-attention mechanism in data correlation learning [50], we build our multi-behavior dependency learning module upon the architecture of multi-head self-attention network, which allows the learned behavior type-specific representations to interact with each other and identify the most informative correlated signals across different types of interaction behavior. Furthermore, considering the fact that different types of interaction behavior (e.g., add-to-cart and purchase) can be mutually correlated in a complex way (due to personalized factors) [2], the multi-head learning strategy enable our behavior dependency encoder with the capability of jointly attending to information from different representation subspaces [48]. In our transformer network, we adopts the scaled dot-product attention for each ℎ-th head with the definitions of query, key and value transformation matrices Q ℎ ∈ R × , K ℎ ∈ R × and V ℎ ∈ R × . Then, the weightˆℎ , ′ assigned to each input value is determined by the dot-product of the query with all the keys as follows: where ℎ , ′ is the intermediate variable fed into the softmax operation to generate the final relevance scoreˆℎ , ′ between the -th and ′ -th type of behavior. Based on the learned head-specific attention weights, our dependency encoding module aims to learns a crosshead relevance score for each behavior type-specific representatioñ X , with the following multi-head learning operations: To alleviate the gradient vanishing issue, the residual connection [9] is employed in the deep neural network structures. Additionally, we element-wisely add the learned dependency-aware behavior type-specific interaction representation Y , with the projected feature embeddingX , of -th behavior, so as to jointly preserve the behavior type-specific interaction features and the underlying interdependent signals across various types of user behavior. Formally, such operation is given as:

Customized Behavioral Context Learning
In addition to the implicit multi-behavior dependency encoded by the above introduced transformer network, each type of behavior may have its own characteristics. For instance, users' page view behavior are more frequent than their purchases and add-to-cart behavior is more likely to be followed by a purchase than the addto-favorite behavior. While the cross-behavior inter-correlation structure can be modeled by our transformer module, the behavior type-specific semantic diversity has been overlooked. Hence, we propose to augment our MATN framework with the capability of capturing the semantic signals of each individual type of interaction behavior. Motivated by the recent advancements of augmented neural architecture and attention mechanism [21,33], we perform a customized representation recalibration process on behavior typespecific context with a memory-augmented attention network. In our memory-based behavior context learning module, we provide a customized transformations for each type of user behavior rep-resentationỸ , by stacking a set of memory blocks. By doing so, we endow MATN with the power of distilling the underlying semantics from the specific contextual user-item interaction scenario (e.g., page view, interested in, want to buy, or purchase). In specific, our customized embedding recalibration module aims to learn (indexed by ) transformation matrices (individual is referred as U ∈ R × ) as the corresponding augmented memory, in order to project the general behavior embeddingỸ , into a type-aware latent learning space. By applying different memory transformations over different types of behavior, each type of behavioral features are refined with respect to its own contexts with the designed memory, and a customized behavioral representations are generated through this type-specific transformation procedure.
Furthermore, in order to alleviate the overfitting phenomenon of type-specific memory augmented neural network architecture [53], Behavior Rep is the dot-product between the embeddings and transformation weight matrix.
we employ an attention network to learn the relations between behavior types and memory matrices in an explicit way, and generate a behavior type-specific transformations with weighted summation. Formally, the refined representation with customized behavioral context for the -th interaction type is where K ∈ R × , b ∈ R are the transformation and bias for calculating attention weights. Instead of using Softmax, we use ReLU to relieve the gradient vanishing issue and mke it easier to train the attentive weight calculating. The memory transformation matrices U and calculating attention weights , are jointly trained with other components of MATN.

Multiplex Relation Aggregation Layer
Next, we build upon a behavior type-wise gating mechanism to aggregate the learned latent representations from the memoryaugmented transformer network, with the exploration of their contributions in capturing user's preference and assisting making predictions on the target behavior. Considering the distinct effects of different types of behavior in characterizing user's interest, e.g. user's historical purchases may be more relevant to his future purchases as compared to his page view activities, the type-specific importance is learned in our gated mechanism in an adaptive manner. Formally, the applied weighted aggregation gate outputs a -dimensional unified representation for as follows: where w ∈ R is the parametric weights for aggregation, the Softmax activation function is used to normalize the weights. By applying the weighted aggregation gate, MATN learns the contributions of different behavior types and thus can enable the adaptive aggregation in modeling the cross-type behavior relations.
After obtaining the aggregated user behavior representation, the MATN adopts a two-layer feed-forward network with non-linear activation, to capture the complex feature interactions in the latent embeddings. Formally the deeply-extracted user representations are learned with the following operation: where W * ∈ R × and b * ∈ R are transformation and bias vectors of the neural network, is element-wisely applying non-linear activation functions, and residual connections are also employed. Γ ∈ R is the final user representation.

The Learning Process of MATN
Given the user behavior representation aggregated from different views (i.e., behavior type-specific semantics and cross-type behavior dependencies), MATN could make predictions on user's preference over items for the target -th type of behavior. In particular, the prediction process is performed through a dot product operation Pr(X , , ) = P ⊤ · Γ , where P ∈ R is from a parametric embedding table for all items, and the result Pr(X , , ) is a scalar score representing 's tendency of interacting with under behavior . Inspired by the settings of learning process on top-N recommendation tasks [23,51], we leverage the pair-wise loss to model the relative position in ranking-based recommendation scenarios. For each training step for user , we sample a positive interaction set { 1 , 2 , ... } composed of interacted items with for the target -th type of behavior. Here. is defined as the number of the positive samples. Correspondingly, the same number of items that have no interactions with in the training set are randomly sampled to form the negative interaction set { 1 , 2 , ..., }. Based on the above descriptions, we formally define our loss function over all the samples of all users as below: where the first term is the pair-wise loss for a positive-negative pair. It expands the signed difference between two predictions, until it  reaches a big enough scale. The latter term is a weight decay regularization term to prevent over-fitting, and is the regularization weight. The learning process is elaborated in Algorithm 1.

EVALUATION
In this section, we perform experiments on different datasets to demonstrate the effectiveness of our MATN. We aim to answer the following research questions: • RQ1: Compared to state-of-the-art models, does MATN achieve better performance in various recommendation applications?
• RQ2: What is the impact of the designed modules in MATN ? Are the proposed cross-behavior transformer network and attention memory module necessary for improving performance?
• RQ3: How is the MATN 's recommendation accuracy w.r.t the integration of different types of behavior?
• RQ4: What is the influence of hyperparameter settings in MATN for the recommendation performance?  platform Yelp; (iii) E-Commerce: a user behavior data from a realworld e-commence platform. Table 1 summarizes the data statistics and we present the data details as below: MovieLens Data 1 . It is a widely-used dataset for performance validation of various recommendation methods. Following the partition strategy in [18,24], we differentiate the explicit user-item interactive behavior into three types in terms of user rating scores (i.e., ranging from 1 (worst) to 5 (best) stars with 0.5 star as increment): the original rating score ≤ 2, > 2 and < 4, ≥ 4 corresponds to the dislike, neutral and like user behavior, respectively. In the MovieLens dataset, we regard the like interaction as the target behavior and other interactions (dislike and neutral) as source behavior, because the positive interactions between users and items may be more useful for capturing user's preferences in recommendations [20].
Yelp Data 2 . This is another recommendation benchmark dataset collected from Yelp. We use the same multi-behavior differentiation strategy as the MovieLens data and partition the 5-star range rating behavior into dislike, neutral and like user behavior. In addition to the user rating behavior, this data includes an additional tip behavior to indicate that user writes a tip on his/her visited venues. Similar to the MovieLens data, the target behavior in Yelp data is also set as the like interaction and others are set as source behavior.
E-Commerce Data. Besides the two benchmark datasets for movie and location-based venue recommendations, we also evaluate our MATN framework in a real-world recommendation scenario with explicit multiple user behavior data from a major online retailing platform. Specifically, this data contains four types of interaction behavior, i.e., page view, add-to-favorite, add-to-cart and purchase. We consider the purchase behavior as the target one, since the purchase is directly related with the conversion rate of recommendation in real-life E-commerce sites [8].

Evaluation Settings and Metrics.
In our experiments, we utilize the leave-one-out evaluation strategy which has been widely utilized in recommendation literature [11,12]. Following their evaluation settings, we regard the latest interaction of each user as the test set and use the rest of data for training. For efficient and fair evaluation, we follow the common strategy in [14,32] to associate each ground truth item with 99 randomly sampled negative instances which have not interacted with the corresponding user.
We leverage two widely-used ranking metrics: Hit Ratio (HR@ ) and Normalized Discounted Cumulative Gain (NDCG@ ) [4,41], to investigate the ranking performance (top-ranked recommended items) of all compared methods. Note that higher HR and NDCG scores reflect better recommendation results. In our experiments, we also evaluate the model performance by varying the value.

Competitive Baselines.
To perform a comprehensive performance validation, we compare our MATN with 12 baselines from six research lines, which are elaborated as follows: Conventional Matrix Factorization-based Recommendation: • BiasMF [15]: This method is built upon the matrix factorization architecture with the incorporation of user and item biases.
Neural Collaborative Filtering Models for Recommendation: • DMF [47]: It is a deep matrix factorization model which takes both the explicit and implicit feedback as the input. • NCF [10]: NCF aims to supercharge collaborative filtering with non-linear neural networks. We consider three variants of NCF w.r.t user-item interaction encoders: i.e., Multilayer perceptron (i.e., NCF-M), concatenated element-wise-product branch (i.e., NCF-N) and the fixed element-wise product (i.e., NCF-G).
Collaborative Filtering with Auto-Encoder: • AutoRec [27]: It leverages a three-layer autoencoder to map user-item interactions into latent representations. • CDAE [45]: In this autoencoder CF, an adaptive loss is incorporated into the embedding projection process for users/items.

Neural Auto-regressive Recommendation Models:
• CF-NADE [53]: It enhances the autoregressive collaborative filtering with the parameter sharing between different ratings. • CF-UIcA [5]: It is a neural co-autoregressive framework to consider the structural correlation for both users and items.

Graph Neural Network Recommendation Models:
• ST-GCN [49]: It stacks encoder-decoder blocks using graph convolutional networks to learn embeddings of users and items. • NGCF [41]: This approach explore the structural knowledge with the message-passing mechanism to capture the high-order connections in the user-item interaction graph.
Recommendation with Multi-Behavior Learning: • NMTR [7]: It is a multi-task recommendation model which considers the behavior correlations in a cascaded manner. • DIPN [8]: This model utilizes bi-directional recurrent network and attention mechanism to consider the correlations between the buying or browsing activities.

Parameter Settings.
In the latent learning space of MATN framework, we set the hidden state dimensionality as 16. In the multi-behavior transformer module, we set the number of attention heads for multi-dimensional learning as 2. Furthermore, the number of memory transformations is set as 8 in our customized behavior-specific context encoding. We implement our MATN with TensorFlow and use Adam optimizer for model optimization with the learning rate and batch size of 1 −3 and 32, respectively. The decay rate of 0.96 is applied for each epoch during the training phase. To reduce the overfitting effect, we adopt set weight decay as the regularization strategy with the selection from {0.05, 0.01, 0.005, 0.001, 0}. The depth of our feature extraction module is set as 3. For the baselines (i.e., NCF and NMTR) which employ the point-wise loss, we set the sampling ratio for positive and negative instances from the range of 1 : 1 to 1 : 4.  Table 2 ("Imp" indicates the relatively improvement ratio). We observe the remarkable performance improvement achieved by our MATN in predicting different types of behaviors. We attribute such improvements to exploration of the cross-type behavior dependencies which are neglected by most existing methods, although they attempt to model complex user-item interactive relations with various deep neural encoders (e.g., autoencoder, graph neural network, attention mechanism). Additionally, by jointly analyzing the results among the three datasets, we find that the improvement of MATN on the E-Commerce data is the most significant with the largest data scale. This may be caused by the behavior diversity: the multiple behaviors from the E-Commerce site are constructed with four different types of behavior which may show strong ordinal relations between the target (purchase) and source behaviors (e.g., page view → add-to-cart → purchase) in the real-world online retailing systems. The consistent improvement across datasets with different user-item interaction densities, suggests the robustness of MATN in accurately learning user preference under different sparsity degrees.
Lastly, it is worth mentioning that although the correlations between behavior has been considered in recent recommendation solutions (i.e., NMTR and DIPN), they merely model the singular dimensional cascading correlations between multi-type interactions, and cannot comprehensively capture the arbitrary dependencies between different types of interaction with different items. Therefore, such oversimplification on the behavior dependency leads to suboptimal recommendation results.

Overall Prediction Click Behavior.
We also conduct experiments to evaluate the recommendation performance of all compared methods by forecasting the overall user-item interaction (i.e., click behavior), since the accurate predictions on overall interactive behavior (e.g., including all page view, add-to-cart and purchase behavior) could also provide useful insights for recommendation scenarios which focus on optimizing the click rate. As shown in Table 3, our MATN still achieves the best performance on all datasets as compared to various types of baselines. This validation shows  Model Metric Click Like @1 @3 @5 @7 @9 @1 @3 @5 @7 @9 We do not utilize the multi-behavior transformer network to capture mutual relations between different types of behavior. • Effect of Memory Attention Mechanism: MATN -M. We remove the memory-augmented attention network in the joint MATN model to encode behavior type-specific semantics. • Effect of Gating Mechanism: MATN -G. We replace the designed gating mechanism with the simplified average pooling operation over all behavior type-specific representations in the behavioral pattern aggregation layer. Figure 2 presents the model ablation study results. We summary the following findings (MATN is the default model version).
(1) The incorporation of mutual dependencies between different types of interaction behavior over all items, is capable of boosting the performance substantially. It demonstrates the rationality of our multi-head self-attention architecture in learning explicit pair-wise relations between different behavior types.
(2) MATN is consistently superior to MATN -M, which hence illustrates the importance of considering context and semantics of individual type of behavior in profiling user preferences.

Impact Studies of Multi-Behavior Relation Integration (RQ3)
To investigate whether exploiting multi-type interaction behavior helps to achieve better performance, we further perform ablation experiments for the purchase prediction task on E-Commerce data, to show the effect of incorporating different types of user-item interactions in our MATN with four model variants: MATN -without the add-to-favorite behavior; MATN -without the add-to-cart behavior; and MATN -without the page view behavior. Furthermore, we design another variant by removing all other types of interactions and only contain the purchase behavior MATN . Figure 3 shows the evaluation results of different variants under varying top-k settings. We summarize the following findings: (1) MATN using all types of interaction behaviors consistently outperforms other variants with varying top-k settings, except for one exception on top-1 prediction with minor performance defect. The results validate that our MATN improve purchase forecasting through integrating multi-behavior relations.
(2) MATN using only purchase data yields worst performance, which shows the positive contribution of the three additional behavior types (i.e. page view, add-to-favorite and add-to-cart) in helping with user modeling in the e-commerce scenario.
(3) Among the three variants that utilize two additional behavior types (i. e. MATN , MATN , MATN ), MATN clearly shows more severe performance degradation compared to the other two. This sheds light on the higher importance and effectiveness of utilizing page view data in MATN and online shopping recommendation.

Hyperparameter Study of MATN (RQ4)
In our experiments, we also investigate the impact of different hyperparameter settings in our developed MATN framework. Specifically, we evaluate the model recommendation performance by varying the values of several key hyperparameters, including the hidden state dimensionality , the memory dimension in our memory attention network, and the number of neural network layers in our deep feature extraction module. The evaluation results on the Yelp data in predicting both click and like behavior are shown in Figure 4. The major findings are summarized as below: • Hidden State Dimensionality . We can observe that when we increase from 4 to 16, the recommendation performance becomes better, but the further increase the value (≥ 32) may not be helpful for the model prediction accuracy. The potential reason for this observation is that the large number of latent units could bring a stronger representation capability.
• Memory Dimension . Our memory attention network enables the behavior type-specific semantics learning could be performed from different dimensions. The parameter study results on memory dimension indicates that performing the transformation with more latent learning sub-spaces will benefit the recommendation at the early stage, but the continuous increase of will lead to the overfitting issue.
• Feature Extraction Network Depth . We further examine whether designing a deep feature extraction network is beneficial to the recommendation task. As we can see, stacking two hidden layers is beneficial to the performance, which is attributed to the high non-linearities brought by more non-linear layers. However, the overfitting phenomenon can be observe when we perform more transformation-based feature interaction operation with more hidden layers (≥ 3).

Case Study on Model Interpretation (RQ5)
In this subsection, we perform qualitative analyses to show the model interpretation of MATN in comprehending user behavior relationships and generate more convincing recommendation. To be specific, we visualize the learned quantitative weights learned by our multi-head self-attention mechanism, memory-augmented attention network and multiplex relation aggregation layer. Four typical cases (i.e., samp 1 ,...,samp 4 ) are sampled from the prediction of overall click behavior and purchase behavior on the E-Commerce dataset. From the visualization results, we have the following observations: (1) page view and purchase behavior could provide more informative signals in predicting the click and purchase, respectively. This make sense since the same type of behavior may share closer relationships than other behavior types. (2) In the head-specific self-attention layer, the 4 × 4 behavior relevance matrix indicates across four types of user behavior. An interesting observation is that: add-to-favorite activity is more like to be correlated with page view and purchase, than add-to-favorite. Similar results can be observed for add-to-cart. It might indicate that add-to-favorite has a high co-occurrence probability than others in the real-world ecommerce platform. (3) The memory attention could learn weights in an adaptive way which corresponds to the importance score generated by our gating mechanism. The reason lies in the utilization of ReLU activation in the attention calculation instead of the mandatory restriction with Softmax function. Overall, all above observations demonstrate the model interpretation power of MATN in capturing complex behavior relations from different perspective.

Scalability Study of MATN (RQ6)
In addition to recommendation accuracy, the model efficiency is also an important factor to investigate. In this subsection, we evaluate the computation time cost of our MATN as compared to other baselines. In Table 5, we report the running time of each epoch during the training phase of each compared approach. We can observe that our MATN model could achieve comparable performance when competing with most baselines, especially in dealing with the large-scale user-item interaction data.
Although we lose in the cases when comparing with some of the competitive baselines-learning user-item interaction representations with simple relation encoders (e.g., Multilayer Perceptron, vanilla autoencoder), our MATN still exhibits competitive model scalability due to the comparable time complexity. However, MATN can show obvious performance superiority over these techniques. In addition, the performance gap (measured by running time) between MATN and graph neural network recommendation methods (i.e., ST-GCN and NGCF), may stem from the high computational cost of graph convolution operation when performing information aggregation and propagation.  Figure 4: Hyper-parameter study in terms of HR@10 and NDCG@10 Self Att.

Memo
Att.  Figure 5: Case study of the learned quantitative weights from key modules in MATN. Pair-wise relations between four types of behavior (e.g., , , and ) are represented with a 4 × 4 weight matrix in the multi-head self-attention layer. 1 ,..., 8 indicate the learned weights across 8 memory dimensions in the memory attention network. The four relevance scores encoded by gating mechanism corresponds to four behavior types. Tuples (e.g., <50,12,39,50>) are numbers of behaviors in the order of (P, F, C, B). Multi-layer Perceptron has been integrated into the collaborative filtering architecture to handle non-linear feature interactions [11,47]. Several work attempts to utilize encoder-decoder network to map explicit user-item interactions into latent representations, using autoencoder [27] and its variants [29]. In addition, another research line lie in leveraging graph neural network to incorporate useritem graph signals into the recommendation framework, such as NGCF [41], STAR-GCN [49] and Multi-GCCF [31]. The major difference between these methods and ours is that MATN explores the cross-behavior interactive knowledge to assist recommendation.
Relation-aware Recommender Systems. Prior work has made significant advances to develop recommender systems with the consideration of various relations between users and items. For example, the social-aware recommender systems aim to boost recommendation performance by exploring user's social relations based on the information dissemination [3,6]. Furthermore, knowledge graph has become another information source from item side to help recommendation models capture relationships between items [37,40]. In addition to the relation of collaborative similarity, there exist work aiming to consider multiple item relationships (e.g., shared director or categories) to learn fine-grained item knowledge [46]. Different from these methods which focus on using the exogenous information from either user or item side, this work explores the multiplicity of pairwise user-item interactions and carefully learns their underlying inter-dependencies.
Attention Network for Recommendations. Attention mechanism has been proven to be effective in differentiating various relations for recommendations [34], such as item transitions [17], user connections [28] and customer group dynamics [36]. To address the limitation of recurrent neural architectures in capturing long-range dependencies without the rigid order assumption, selfattention mechanism has been introduced to model correlations from any pair of positions of input data points [35]. For example, Sun et al. [30] proposed a bidirectional self-attention framework for sequential recommendation. Additionally, multi-head self-attentive model is introduced to recommended news to users [43]. Our MATN framework is motivated by the multi-head self-attentive learning architecture in a sense that a memory augmented transformer is designed to model multiplex behavior relation dynamics from different types of user-item interactions.

CONCLUSION
In this work, we propose MATN, a novel memory augmented transformer neural architecture which incorporates multiple types of user behavior relationships into a cross-behavior collaborative filtering framework. We argue that these different types of user-item interactions are usually neglected in conventional methods. MATN demonstrates the state-of-the-art performance on two benchmark datasets and a large-scale user behavior data from a major online retailing platform. In addition, via the qualitative analysis of the attentive weights, we discover that the implicit cross-type behavioral dependencies are encoded within the MATN framework. Notwithstanding the interesting problem and promising results, some directions exist for future work. We will next incorporate rich auxiliary data source (e.g., user review text information or item description [52]) to further enhance the current recommendation framework. Additionally, another time dimension of the problem deserves more investigation. When multi-type user-item interaction data arrives in a timely manner, how to best account for it in the current MATN framework? One possible direction is adapting MATN to a time-sensitive model by analyzing the trade-off between accuracy and complexity.