{'fraction_non_alphanumeric': 0.035822460391425906, 'fraction_numerical': 0.011824324324324325, 'mean_word_length': 4.950953206239168, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Sequential Recommendation with Bidirectional Chronological Augmentation of Transformer

Sequential recommendation can capture user chronological preferences from their historical behaviors, yet the learning of short sequences (cold-start problem) in many benchmark datasets is still an open challenge. Recently, data augmentation with pseudo-prior items generated by Transformers has drawn considerable attention. These methods can generate pseudo-prior items sequentially in reverse chronological order to extend the original sequences. Nevertheless, the performance may still dramatically degrade in very short sequences; most notably, the generation of pseudo-prior items does not take into account the forward direction (from the past to the future), and so the underlying temporal correlations are not preserved in terms of conditional probabilities. Motivated by this, we propose a Bidirectional Chronological Augmentation of Transformer (BiCAT) that uses a forward learning constraint in the reverse generative process to capture contextual information more effectively. Then, self-knowledge distillation is adopted in augmented and original sequences to bridge the gap between data augmentation and model representation, which enhances the robustness of sequence encoder. Moreover, an informative positive and negative sampling strategy is proposed to accelerate optimization and prevent overfitting. Extensive experiments on two popular real-world datasets demonstrate the efficacy of our method on very short sequences (L<3) and long sequences (20<L<50) as well, our approach outperforms state-of-the-art models by an average of 35.04% and 8.76% respectively, in terms of Recall@5. Source code is available at https://github.com/juyongjiang/BiCAT.


INTRODUCTION
Recommender systems are a key engine underlying B2C commercial platforms such as the YouTube video and the Amazon shopping recommendation. Sequential recommendation, in particular, represents recommender systems that learn from historical user behaviors to capture the sequential preference and infer the next possible item [25,52]. A fundamental feature of sequential recommendation is the sequential order of user actions, which characterizes the influence of past behaviors on the current and future ones [4,29]. Based on this, transformer models such as SASRec [18] and BERT4Rec [40] have shown extraordinary performance in many sequential recommendation tasks.
Despite the success of state-of-the-art transformers in the sequential recommendation, researchers have reported that they are not effective on short sequences [27]. With limited information about the user sequential patterns, short sequences are much harder to predict, which is also known as the cold-start issue [20,44]. Yet in the meantime, short sequences can be dominant in real-world datasets. As shown in the exemplified histogram of the Amazon Beauty dataset in Figure 1, 96% sequences have less than 20 items and only 0.6% sequences have more than 50 items.
To address this problem, data augmentation methods [21,46,53] are typically adopted to improve the model performance. Early augmentations methods [49,58] such as mask, replace, crop, and add may break the original sequential property, as shown in Figure  2. Therefore, it could exaggerate the cold-start issues since short sequences are vulnerable to the slight change of a few items [26].
Recently, ASReP [27] was proposed to learn the reverse sequential correlation (i.e., future-to-past) from data to generate pseudoprior items at the beginning of the original sequences. Given the length-augmented sequences, it then fine-tunes the transformer from the original direction (i.e., past-to-future) to predict the next items. We illustrate the Recall@5 for the next-item prediction on Amazon Beauty dataset with and without ASReP in Figure 1. As noted, recommendation performance on short sequences is worse than on long sequences. Although ASReP can improve the learning of short sequences, the performance on short sequences is still worse than on long sequences. ASReP has an advantage over other methods in that it does not change the original sequences and only fills in the prior "missing" items. However, the augmentation process is a backward process (future to past), so the temporal correlations underlying the generated sequence may not be fully consistent with the original sequence in terms of the forward direction (past to future). As shown in Figure  3, although the reverse learning can maximize the reverse probability, the generated item may not fit in the original sequential correlation to maximize the forward probability to discover the future ground-truth item . To illustrate the reasoning, we provide theoretical explanations in the methodology section with a counterexample in Figure 4.
Since ASReP may not fit in the correlation of the forward recommendation step, to preserve the sequential correlation in the augmentation process, we propose a Bidirectional Chronological Augmentation of Transformer (BiCAT) that uses a forward learning constraint in the loss to capture the contextual information when generating items reversely. Namely, when using the newly generated items to predict their subsequent items, such an extra learning condition in the augmentation phase is more consistent with the forward recommendation, which we believe could lead to better generalization performance. In the meantime, BiCAT can also serve as a better pre-training for the recommendation task.
Besides, to transfer more contextual information of augmented sequences to non-augmented sequences, we employ self-knowledge distillation between augmented and original sequences to bridge the gap between data augmentation and model representation and enhance the robustness of sequence encoders. Moreover, we notice that most state-of-the-art methods build on SASRec [18] commonly adopt the same positive and negative sampling at the training stage. It neglects the problem that the early time steps of positive and negative items will impair the optimization of sequence encoders due to narrow sequential information, which is also known as the cold start problem. To this end, we clip the part of early time step items in each positive and negative sequence to accelerate model optimization and prevent overfitting. We verify the effectiveness of our BiCAT by reproducible experimental results on two real-world datasets. Comparisons to multiple baseline models show that BiCAT can improve the learning of not only short sequences but also long sequences by a large margin. The main contributions of this paper are: • We propose Bidirectional Chronological Augmentation of Transformer (BiCAT) that adds a novel forward learning constraint to preserve the original correlation when generating items reversely, which bridges the gap between augmentation and recommendation, and prove it theoretically. • We leverage self-knowledge distillation between augmented and original sequences to bridge the gap between data augmentation and model representation, and enhance the robustness of sequence encoders. • We notice a serious sampling problem existed in most SASRecbased models that the early time step of positive and negative items will impair the model optimization, thus informative positive and negative sampling strategy is proposed. • We give extensive experiments to show that BiCAT improves the transformer on public datasets significantly. For short sequences with lengths smaller than 3 in particular, BiCAT can improve the transformer by 35.04% on average and improve the ASReP augmented model by 10.19% on average, both in terms of Recall@5. In addition, we released our codes, datasets, and the baselines 1 .

RELATED WORK 2.1 Sequential Recommendation
The core of sequential recommendation is the modeling of user-item interaction pattern and the prediction of future items based on each user's historical behaviors. Early works [9,19,36] mainly reply on Markov chains to model the pair-wise item transition. Later, with the rise of deep learning, Markov models are extensively replaced by neural networks. Recurrent neural networks (RNNs) [13,25,52] that can store the sequential memory are widely used in sequential recommendations. RNNs with invariant structures and/or gating mechanisms [29,34,39] are also proposed for different recommendation tasks. Other improvements for RNNs are also studied, such as using contextual features [38] and alternative sampling tricks [12]. Convolutional neural networks (CNNs) [42,54] are also proposed to focus on learning the local transitions. These models with limited reception fields still faces the long-term dependency problem. More recently, attention-based models [16,18,28,31,40] are adopted in sequential recommendation tasks, which achieve outstanding performance. For example, SASRec applies the transformer layer from NLP tasks to adapt to sequential recommendation. BERT4Rec further considers a bidirectional transformer layer. Additionally, other techniques such as relation awareness [15], graph learning [30] and reinforcement learning [50] are also used to adapt to various sequential recommendation tasks.

Data Augmentation
Data augmentation is a useful component in machine learning venues to increase the volume of data and hence the model performance. In the literature of recommender systems, many kinds of data augmentation methods are proposed. For example, [41] uses data augmentation of sequence preprocessing and embedding dropout to enhance training and reduce overfitting. [53] proposes to integrate future interaction as a data augmentation to improve model training. Generative augmentations [23,45] are proposed to add extra user-item interaction data by generative adversarial networks or Seq2Seq models. A rising technique that can be classified as data augmentation is self-supervised learning for sequential recommendation [26,49,51,58], which uses operations such as crop, reorder, mask, substitute, and insert to supplement the original sequences. These methods are designed for improving model robustness, but may not address the cold-start problems since they may inevitably break the sequential property of the original sequences and the learning of short sequences is vulnerable to the change of a few items [26]. Recently, a reverse data augmentation method ASReP [27] is proposed for the cold-start learning of short sequences, which adds pseudo-prior items in the beginning of the original sequences by learning to predict the past items reversely. Traditionally, cold-start problems are mainly addressed by using side information [37], knowledge transfer [17], and meta-learning [3,56].

Self-Knowledge Distillation
Knowledge distributions [1,5,6,14,24,57] aims to maximize the consistency of representation capability of two different models, where the two models refer to the teacher (big and complex) and student (small and simple), respectively. In our work, the teacher and student are the same models but with augmented and original sequential input data, respectively, thus it resembles the selfknowledge distillation [33] scenario. Besides, Bidirectional KLdivergence is leveraged to minimize the output distributions of two sub-models sampled by different input data. It is worthwhile to note that different from the existing method which exploits dark knowledge from the model itself [7,8] or distills knowledge from the randomness of sub-models brought by dropout [47] or between different layers [55], our approach can be regarded as an input-wise self-knowledge distillation without extra parameters and dropout settings. In other words, each pair of sampled sub-models perform distillation between each other for augmented and original input.

THEORETICAL ANALYSIS
We prove by Bayes' theorem that the forward sequential correlation is not necessarily consistent with the reverse sequential correlation, so the reverse augmentation may lead to sub-optimal augmentation. Assume that and are two items interacting with a user chronologically. Since the user interacts with first, we denote the forward correlation as ( | ) and write ( | ) = where ( ) ≠ 0. Thus, the reverse correlation ( | ) equals the forward correlation ( | ) only when ( ) = ( ). It indicates that the two items always appear concurrently, which is not always true in practical item sequences. We illustrate this with an example in Figure 4. We define three events (i.e., , , ), and two probabilities (i.e., forward probability and reverse probability ). We consider four sequences as the training data in Figure 4 (a) to indicate the various inference probabilities of the model. For instance, to calculate the forward correlation ( | ), we count the cases with on the right side and on the left side (i.e., "CA" in the forward order) and divide it by the number of times with on the right (i.e., "XA" where X can be any events). In this case, ( | ) = ( , )/ ( ) = 3/4 with three "CA" and one "CD" as shown in Figure 4 Now, if we use to predict the next item in the forward recommendation task, what would be the best choice for augmenting with a prior (left) item? By reverse inference, ( | ) = 3/4 is the largest among other possibilities, so the reverse augmentation model should choose as shown in Figure 4 (b)(ii.). However, does improve the model performance? By forward inference on top of an augmented "BC", we find that ( | ) = 2/3 < 3/4, which is, in fact, worse than the original performance. This counterexample shows that reverse augmentation does not always improve the learning of the original sequential correlation.
In light of this challenge, we can also consider a different constraint that chooses the left element by maximizing the forward correlation instead of the reverse correlation. As shown in Figure  4 (b)(iii.), we find that can achieve that and make the forward recommendation more accurate with ( | ) = 1/1 > 3/4. This example shows that the forward-constrained augmentation can effectively preserve of the original forward sequential correlation. We formally introduce our method in the following sections.

METHODOLOGY 4.1 Problem Statement
denotes the interaction sequence in chronological order of user ∈ U. Each ( ) refers to an item in V that has interacted with at time step . Sequential recommendation aims to predict the item that user will interact with at the ( + 1)th step. The probability of each item ∈ V becoming the ( + 1)th item is denoted as ( ( ) +1 = | ), given the training sequence .

Model Architecture
Here, we introduce an effective data augmentation model of pseudoprior items for short sequences, termed Bidirectional Chronological Augmentation of Transformer (BiCAT). It is built upon the prior work [27] and [18]. The proposed BiCAT consist of 1) an embedding layer; 2) multi-stacked unidirectional transformer layers; and 3) a bidirectional chronological pre-trained transformer for short sequences augmentation. With these modules, the left-to-right transformer is fine-tuned using the augmented sequences for the next item prediction. Our method employs a forward learning constraint in data augmentation to capture the contextual information when generating items reversely. The model architecture is shown in Figure 5.

Embedding Layer
In standard Transformer [43], the embedding layer consists of item embedding and positional embedding. The item embedding layer encodes each item ID ∈ R 1 into a latent representation ∈ R . The item embedding transforms the scalar item IDs into dense vectors, aiming to reduce computation and improve representation learning. To make a fair comparison, we follow [27] to hold a position embedding table ∈ R × , where is the maximum sequence length and is the same embedding dimensions of the item embedding. Note that for a sequence , we truncate it to be the last items if its length is longer than or utilize zero-padding if its length is shorter than [18]. Overall, the embedding layer transforms the sequence of S = [

Transformer Layer
As shown in Figure 5 (a), the basic Transformer layer Trm is composed of two sub-layers, a Multi-Head Self-Attention sub-layer and a Position-wise Feed-Forward Network.

4.4.1
Multi-Head Self-Attention. Specifically, the Multi-Head Self-Attention sub-layer adopts a multi-head attention layer and then a normalization operation on the embedded sequence. Each attention layer adopts the scaled dot-product attention layer used in [18,27,40,43,48], which potentially learn the different importance pattern of items in sequences from ℎ sub-spaces. where Here, = ( ) ∈ R × is the embedded interaction sequence. Note that we omit the layer subscript for convenience. ℎ denotes the number of attention layers.
∈ R × denote four different and learnable parameters that process the Q, K, V, and the output of the multi-head attention layer. The effect of temperature √ is to avoid overly large values of the inner product and extremely small gradients [14,18,40,43]. The tensor dimension of the final output of the multi-head attention sub-layer is still maintained as R × .

Position-wise Feed-Forward Network.
To enhance the representation learning capability, we use a Position-wise Feed-Forward Network to project the output of the multi-head attention sub-layer to the sequence embedding at each position . Specifically, the FFN contains two affine transformations layers with a ReLU activation function, as used in all the prior works [18,27,43].

Bidirectional Chronological Pre-training
We use the transformer in SASRec [18] as our backbone, as shown in the left-hand side of Figure 5. First, we reverse the original Then, we inject the ← into the Transformer to generate the next items as the pseudo-prior items for short sequences. This reversely generation (inference) can be formulated by: where 0 ∈ V denotes the pseudo-prior items. In the training phase,  Secondly, short sequences augmentation is to generate pseudo-prior items for short sequences. Finally, augmented and original sequences are fed to fine-tuning the model for next item ( +1 ) prediction with distributed regularization loss.
the Transformer can capture the reverse item correlations [27,40] and learn a reversely chronological user's preferences. However, as shown in Figures 3 and 4, the reverse augmentation may be inconsistent with the original correlation. Herein, we propose the forward learning constraint, which bridges the gap between reverse augmentation and forward recommendation. Specifically, the forward constraint leverages the generated pseudoprior item 0 to predict item in the forward direction in parallel with Eq. (6). Formally, we feed the augmented sequence where ∈ V denotes the future items of the original sequence at position . Similarly, we also mask the right items at position in → by zeros and train the Transformer to predict them. Since we unearth Bi-Chronological user's preferences from their historical behaviors simultaneously, we name our augmentation model as BiCAT in short.

Prediction Layer.
After layers adaptively and hierarchically aggregate information across all positions, we get the final output H ∈ R × for all items of the input sequence. Then we apply an matrix factorization (MF) [19] layer to predict the relevance of item : where , is the relevance of item being the next item given the first items (i.e., 1 , 2 , ..., ), and N table ∈ R × | V | denotes an item embedding table. Note that we use shared item embedding table in the input and output layer.

Network
Training. With the bidirectional chronological losses, the model has a stronger learning criterion to achieve a better performance. In this work, following SASRec [18], we adopt the binary Here, is a hyper-parameter to balance the forward constraint and the reverse generation. It is set as 1 by default. Our model takes

Short Sequence Augmentation.
After the training of the bidirectional chronological pre-trained Transformer, following [27], we use it to recursively generate pseudo-prior items, as illustrated in Figure 5 (b). To be specific, we use to denote the number of augmented items and to denote the length threshold of whether a sequence is a "short sequence". Formally, we indicate the augmented pseudo-prior items as [

Informative Positive and Negative Sampling
Most state-of-the-art methods build on SASRec [18] adopt the same sampling strategy that randomly generates one positive and negative item for each time step in each sequence at the training stage. However, it is ignored that the early time step of positive and negative items will impair the optimization of sequence encoders due to narrow sequential information, which is also well known as the cold start problem, leading to sub-optimal sequence encoders. To this end, a simple yet effective positive and negative sampling is proposed, which is to clip the part of early time step items in each positive and negative sequence for training. This prevents the adverse effects of low-quality early items on learning that are difficult to optimize. The number of clipped items is denoted by .

Left-to-Right Fine-tuning
Finally, we fine-tune the pre-trained transformer with the augmented S and original S sequences to predict the next item +1 in the left-to-right direction. Specifically, at each training step, we feed S and S to go through the forward pass of the network. Subsequently, we can obtain two distributions of the model predictions, denoted as P 1 (S ; ) and P 2 (S ; ) in which represents model parameters. For the different input data, the distributions of P 1 (S ; ) and P 2 (S ; ) are different. Then, at this training step, our BiCAT method tries to regularize the model predictions by minimizing the bidirectional Kullback-Leibler (KL) divergence between these two different output distributions, which is: With the basic binary cross entropy loss L , see Equation (9), for the next item prediction. The final training objective is to minimize L for sequential data S and S : where is the coefficient weight to control the importance of L .
Input : Training data S and S Output : The model parameter . 1 Initialize model with pre-trained BiCAT. 2 while not converged do 3 randomly sample mini-batch data from S and S , 4 input data twice and obtain the output distribution P 1 (S ; ) and P 2 (S ; ), 5 calculate the binary cross entropy loss L by Equation (9), 6 calculate the KL-divergence loss L by Equation (12), 7 update the model parameters by minimizing loss L of Equation (13). 8

end
In this way, our BiCAT regularizes the model representation space beyond data augmentation and improves the generalization ability of model. The left-to-right fine-tuning of BiCAT is presented in Algorithm 2.
Note that the forward learning constraint is only used in the pseudo-prior item generation stage, not in the fine-tuning stage. The fine-tuning of our BiCAT is more efficient and effective than ASReP [27], indicating that our pre-trained transformer learns forward sequential pattern that is in line with the recommendation task.

Discussion. Augmented sequences S
contains more information related to user's interest pattern by pseudo-prior items so that model could predict the next item more easily and precisely. But we argue that it is non-trivial to inject non-augmented sequences S (i.e. original sequences) into model training since it can bridge the gap between data augmentation and model representation and further enhance the model adaptability. More convincing evidences will be demonstrated in our ablation study.

EXPERIMENTS
In this section, we present our experimental setup and evaluate the BiCAT on two public benchmarks. We answer the following Research Questions (RQs): • RQ1: Does BiCAT outperform the state-of-the-art sequential recommendation models to demonstrate the efficacy and necessity of reverse data augmentation? • RQ2: How does the BiCAT perform in particular on short ( ≤ 3) and long (20 < ≤ 50) sequences? • RQ3: What is the sensitivity of core hyperparameters in the BiCAT? How to set hyperparameters properly? • RQ4: How does BiCAT perform compared to other models in the left-to-right training stage? • RQ5: Ablation study of the influence of various component designs in the BiCAT architecture. • RQ6: How does the BiCAT perform with data augmentation methods for sequential recommendation, i.e., random mask, random delete, random replace, and random add?

Datasets
In this paper, our main contribution is the more effective data augmentation. To compare fairly with prior works to show the effectiveness of our method, we use the same datasets as used in [27]. Amazon 2 is a series of datasets introduced in [18] which consist of large corpora of product reviews crawled from Amazon.com by McAuley et al. The data is split as separate datasets according to the top-level product categories on Amazon. In our experiments, we use the Amazon Beauty (5-core) and Amazon Cell Phones and Accessories (5-core) [32]. We followed the common data preprocessing procedure from [18,27,40]. We treat each rating or review as a presence of a user-item interaction, group the interactions by user ids, and sort them by timestamps to form a sequence for each user. Note that more than 75% sequence length are less than 9 and 7 for beauty and phones, respectively.

Experimental Settings
To evaluate all models, we adopted the widely used leave-one-out evaluation evaluation task [11,18,27,40] to evaluate the sequential recommendation models. For each user, we use the last item of each use for testing, the second last item for validation, and the remaining items for training. Moreover, we follow the common practice in [2,18,22,27,40] to pair each ground-truth item in the test set with 100 randomly sampled negative items that are not interacted by the user. We employ three evaluation metrics: Recall (HR), and Normalized Discounted Cumulative Gain (NDCG) and Mean Reciprocal Rank (MRR). The reported results are averaged for ten random seeds. Codes to reproduce the results are uploaded.

Baselines & Implementation Details
For the two transformer-based models SASRec [18] and BERT4Rec [40], and the two static models BPR-MF [35] and LightGCN [10], we cite the experimental results from ASReP [27]. To implement ASReP and its re-training (RT) variants, we use code provided by the authors 3 . To ensure fairness, our method's implementation is based on their codes and only adds our core component on it.
Additionally, we also create the re-training (RT) variants of the BiCAT, which means training a new Transformer from scratch using      (1) The two variants RT-ASReP and RT-BiCAT are worse than ASReP and BiCAT, which verifies that pre-training is useful for sequential recommendations. (2) RT-BiCAT and BiCAT outperform ASReP in all datasets on all evaluation metrics. The improvement is based on the learning of bi-chronological item correlation to generate higher-quality pseudo-prior items for forward recommendation task.

Overall Performance Comparison (RQ1)
The comparisons and analyses show that the proposed augmentation and pre-training techniques in our BiCAT can significantly benefit sequential recommendation models.

Performance on Short and Long Sequences (RQ2)
We discuss the effectiveness of BiCAT on short and long sequences in the following and report the results in Table 2 and 3.
• Short Sequence Learning. For short sequences with lengths smaller than 3, BiCAT improves SASRec by an average increase of 40.46% on recall rates and ASReP by an average increase of 11.85% on recall rates on both datasets. This shows that the augmentation techniques can work on short sequences and BiCAT provides a significant increase in Recall, NDCG, and MRR over previous state-of-the-art models. • Long Sequence Learning. For long sequences with lengths from 20 to 50, BiCAT improves SASRec by an average increase of 11.08% on recall rates and ASReP by an average increase of 10.59% on recall rates on both datasets. This reveals that while ASReP can hardly work on long sequences, BiCAT can still provide significant improvement in the model recommendation performance for long sequences.
In general, BiCAT outperforms other baselines on both long and short sequences on all six metrics. These results demonstrate that our proposed reverse data augmentation of pseudo-prior items can be a general method to improve recommendation performance, not only limited to short sequences as discussed by most prior works. It verifies that our method can generate higher-quality pseudo-prior items that are beneficial for the sequential recommendation and can apply to a wider range of application scenarios.

Loss Beauty Phones
Loss Epoch Epoch Figure 6: Fine-tuning loss visualization. The blue dash curves denote ASReP, the green dot curves denote SASRec, and the red solid curves denote BiCAT.

Hyperparameter Study (RQ3)
First, we analyze the effect of the two core hyper-parameters and , which indicate the number of pseudo-prior items and the threshold for short sequences respectively, and report the overall performance results on all sequences in Table 4. For each result, we only change the current one hyper-parameter but keep others the same as the optimal settings. We find that the performance reaches a peak on both datasets when and range from 15 to 18. Specifically, for Beauty datasets, the optimal setting is = 15 and = 18; for Phones datasets, the optimal setting is = 17 and = 18. The analysis generally suggests that the proper selection of hyperparameters is quite important to ensure a good performance. and should not be set too small. We also show the effect of other hyper-parameters including , , and in Figure 7. We find that the model performance is slightly dependent on the selection of hyperparameters in most cases but the sensitivity varies in different datasets. For example, our model performance on Phones dataset can hardly be influenced by and , but can be affected drastically by ≥ 0.3. On Beauty dataset, all three hyperparameters may have a slight effect on the model performance, especially when they are set too small.

Fine-Tuning Performance (RQ4)
To further verify our motivation that BiCAT can bridge the gap from augmentation to recommendation and provide a better pre-trained model, we visualize the fine-tuning training loss. As shown in Figure  6, the loss curve of BiCAT falls in between ASReP and SASRec, while BiCAT provides the best recommendation performance. That shows that ASReP is probably more prone to overfitting due to not being in line with the left-to-right recommendation task.

Ablation Study (RQ5)
In Table 5, we conduct ablation study to investigate the contribution of each proposed component of BiCAT on Beauty and Phones datasets in terms of Recall@5 and NDCG@5. In particular, the BiCAT variant without Bi (Bidirectional Chronological Augmentation), AUG (Data Augmentation with pseudo-prior items), KL (Bidirectional Kullback-Leibler), and IS (Informative Sampling) shown in the first row will be degenerated to SASRec. Besides, ASReP will be equivalent to the variant in the second row with AUG and without Bi, KL, and IS. We find that: • AUG remarkably increases evaluation metrics compared with SASRec by 22.19% on average. This demonstrates that AUG is very effective for sequential recommendation model. • Bi and IS can improve the performance of SASRec by 25.49% on average. This verifies that the proposed reverse learning constraint and informative sampling in BiCAT are effective. • Minimizing bidirectional Kullback-Leibler (KL) divergence between S and S output distributions in BiCAT can significantly improve performance. KL technique is much more effective on BiCAT than on ASReP. These verify the efficacy of self-knowledge distillation to bridge the gap between data augmentation and model representation and show that the design of forward learning constraint in BiCAT is essential for KL and IS techniques to work.

Comparison with other augmentation methods (RQ6)
In this section, we compare the recommendation performance of BiCAT on Recall@10 with other data augmentation methods in the literature, including Mask, Crop, Replace, Add, and ASReP. All these data augmentation methods use SASRec as the backbone model. As shown in Figure 8, most conventional methods like Mask, Crop, Replace, and Add make the recommendation performance of SASRec worse on Beauty and Phones datasets, as also reported by some prior works [26,27]. This is probably because operations like Mask and Add change original data randomly and inevitably break the sequential property of the original sequences. ASReP improves SASRec effectively, but BiCAT performs even better as a state-ofthe-art data augmentation method for sequential recommendation.

CONCLUSION
In this paper, we propose a new state-of-the-art model for sequential recommendation with data augmentation. Data augmentation of pseudo-prior items as a useful method can enhance the learning of short sequences to address the cold-start problem. We prove theoretically that the reverse augmentation ASReP may not be consistent with the forward sequential correlation. To generate higher-quality items, we propose BiCAT which uses a forward learning constraint to capture the contextual information when generating items reversely. Since the forward constraint is in line with the sequential recommendation task, it can bridge the gap between reverse augmentation and forward recommendation. Besides, the self-knowledge distillation between augmented and original sequences is leveraged to bridge the gap between data augmentation and model representation. Moreover, an informative positive and negative sampling strategy is proposed to accelerate model optimization and prevent overfitting. Extensive experiments show the superiority of BiCAT by significant improvement over the stateof-the-art ASReP and SASRec not only on short sequences but also on long sequences as a general framework. What's more, our Bi-CAT can conduct a better pre-trained Transformer to help the main task quickly converge at the fine-tuning stage. For future studies, the generalization of BiCAT may also apply to other tasks such as video and language generation. How to generate dynamic items with respect to timestamps also pends future works.