{'fraction_non_alphanumeric': 0.035104922533830164, 'fraction_numerical': 0.011472837811335556, 'mean_word_length': 4.773730892621249, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Temporal Pattern Attention for Multivariate Time Series Forecasting

Forecasting multivariate time series data, such as prediction of electricity consumption, solar power production, and polyphonic piano pieces, has numerous valuable applications. However, complex and non-linear interdependencies between time steps and series complicate the task. To obtain accurate prediction, it is crucial to model long-term dependency in time series data, which can be achieved to some good extent by recurrent neural network (RNN) with attention mechanism. Typical attention mechanism reviews the information at each previous time step and selects the relevant information to help generate the outputs, but it fails to capture the temporal patterns across multiple time steps. In this paper, we propose to use a set of filters to extract time-invariant temporal patterns, which is similar to transforming time series data into its"frequency domain". Then we proposed a novel attention mechanism to select relevant time series, and use its"frequency domain"information for forecasting. We applied the proposed model on several real-world tasks and achieved state-of-the-art performance in all of them with only one exception.

Historical prices of crude oil, gasoline, and lumber. Units are omitted and scales are normalized for simplicity.

Introduction
In everyday life, time series data are everywhere. We observe evolving variables generated from sensors over discrete time steps and organize them into time series data. For example, household electricity consumption, road occupancy rate, currency exchange rate, solar power production, and even music notes can all be seen as time series data. In most cases, the collected data are often multivariate time series (MTS) data, such as the electricity consumption of multiple clients, which are tracked by the local power company. There can exist complex dynamic interdependencies between different series that are significant but difficult to capture and analyze.
Analysts often seek to forecast the future based on historical data. The better the interdependencies among different series are modeled, the more accurate the forecasting can be. For instance, as shown in Figure 1 1 , the price of crude oil heavily influences the price of gasoline, but has a smaller influence on the price of lumber. Thus, given the realization that gasoline is produced from crude oil and lumber is not, we can use the price of crude oil to predict the price of gasoline.
In machine learning, we want the model to automatically learn such interdependencies from data. Machine learning has been applied to time series analysis for both classification and forecasting [G. Zhang andHu(1998), Zhang(2003), Lai et al.(2018) Lai, Chang, Yang, and Liu, Qin et al.(2017) Qin, Song, Cheng, Cheng, Jiang, and Cottrell]. In classification, the machine learns to assign a label to a time series, for instance evaluating a patient's diagnostic categories by reading values from medical sensors. In forecasting, the machine predicts future time series based on past observed data. For example, precipitation in the next days, weeks, or months can be forecast according to historical measurements. The further ahead we attempt to forecast, the harder it is.
In this paper, we propose the temporal pattern attention, a new attention mechanism for MTS forecasting, where we use the term "temporal pattern" to refer to any time-invariant pattern across multiple time steps. The typical attention mechanism identifies the time steps relevant to the prediction, and extracts the information from these time steps, which poses obvious limitations for MTS prediction. Consider the example in Figure 1. To predict the value of gasoline, the machine must learn to focus on "crude oil" and ignore "lumber". In temporal pattern attention, instead of selecting the relevant time steps as in the typical attention mechanism, the machine learns to select the relevant time series.
In addition, time series data often entails noticeable periodic temporal patterns, which are critical for prediction. However, the periodic patterns spanning multiple time steps are difficult for the typical attention mechanism to identify, as it usually focuses only on a few time steps. In temporal pattern attention, we introduce a convolutional neural network (CNN) [LeCun and Bengio(1995), A. Krizhevsky and Hinton(2012)] to extract temporal pattern information from each individual variable.
The main contributions of this paper are summarized as follows: -We introduce a new attention concept in which we select the relevant variables as opposed to the relevant time steps. The method is simple and general to apply on RNN. -We use toy examples to verify that our attention mechanism enables the model to extract temporal pattern and focus on different time steps for different time series. -Attested by experimental results on real-world data ranging from periodic and partially linear to non-periodic and non-linear tasks, we show that the proposed attention mechanism achieves state-of-the-art results across multiple datasets. -The learned CNN filters in our attention mechanism demonstrate interesting and interpretable behavior.
The remainder of this paper is organized as follows. In Section 2 we review related work and in Section 3 we describe background knowledge. Then, in Section 4 we describe the proposed attention mechanism, after which we present and analyze the experimental results in Section 6. We conclude in Section 7. The most well-known model for linear univariate time series forecasting is the autoregressive integrated moving average (ARIMA) [G. E. Box and Ljung(2015)], which encompasses other autoregressive time series models, including autoregression (AR), moving average (MA), and autoregressive moving average (ARMA). Additionally, linear support vector regression (SVR) [Cao and Tay(2003), Kim(2003)] treats the forecasting problem as a typical regression problem with time-varying parameters. However, these models are mostly limited to linear univariate time series and do not scale well to MTS. To forecast MTS data, vector autoregression (VAR), a generalization of AR-based models, was proposed. VAR is probably the most well-known model in MTS forecasting. Nevertheless, neither AR-based nor VAR-based models capture non-linearity. For that reason, substantial effort has been put into non-linear models for time series forecasting based on kernel methods [Chen et al.(2008)Chen, Wang, and Harris], ensembles [Bouchachia and Bouchachia(2008)], or Gaussian processes [Frigola and Rasmussen(2014)]. Still, these approaches apply predetermined non-linearities and may fail to recognize different forms of non-linearity for different MTS.
Recently, deep neural networks have received a great amount of attention due to their ability to capture non-linear interdependencies. Long shortterm memory (LSTM) [Hochreiter and Schmidhuber(1997)], a variant of recurrent neural network, has shown promising results in several NLP tasks and has also been employed for MTS forecasting. Work in this area began with using naive RNN [J. Connor and Martin(1991)], improved with hybrid models that combined ARIMA and multilayer perceptrons [G. Zhang andHu(1998), Zhang(2003), Jain and Kumar(2007)], and then most recently progressed to dynamic Boltzmann machines with RNN [Dasgupta and Osogami(2017)]. Although these models can be applied to MTS, they mainly target univariate or bivariate time series.
To the best of our knowledge, the long-and short-term time-series network (LSTNet) [Lai et al.(2018)Lai, Chang, Yang, and Liu] is the first model designed specifically for MTS forecasting with up to hundreds of time series. LSTNet uses CNNs to capture short-term patterns, and LSTM or GRU for memorizing relatively long-term patterns. In practice, however, LSTM and GRU cannot memorize very long-term interdependencies due to training instability and the gradient vanishing problem. To address this, LSTNet adds either a recurrent-skip layer or a typical attention mechanism. Also part of the overall model is traditional autoregression, which helps to mitigate the scale insensitivity of neural networks. Nonetheless, LSTNet has two major shortcomings when compared to our proposed attention mechanism: (1) the skip length of the recurrent-skip layer must be manually tuned, whereas the proposed approach learns the periodic patterns by itself; and (2) the LSTNet model is specifically designed for MTS data with strong periodic patterns, whereas the proposed attention mechanism, as shown in our experiments, is simple and adaptable to various datasets, even non-periodic and non-linear ones.

Preliminaries
In this section, we briefly describe two essential modules related to our proposed model: the RNN module, and the typical attention mechanism.

Recurrent Neural Networks
Given a sequence of information {x 1 , x 2 , . . . , x t }, where x i ∈ R n , an RNN generally defines a recurrent function, F , and calculates h t ∈ R m for each time step, t, as where the implementation of function F depends on what kind of RNN cell is used. Long short-term memory (LSTM) [Hochreiter and Schmidhuber(1997)] cells are widely used, which have a slightly different recurrent function: which is defined by the following equations: where i t , f t , and o t ∈ R m , W xi , W x f , W xo and W xg ∈ R m×n , W hi , W h f , W ho and W hg ∈ R m×m , and denotes element-wise multiplication.

Typical Attention Mechanism
In the typical attention mechanism [Luong et al.(2015)Luong, Pham, and Manning, Bahdanau et al.(2015)Bahdanau, Cho, and Bengio] in an RNN, given the previous states H = {h 1 , h 2 , . . . , h t−1 }, a context vector v t is extracted from the previous states. v t is a weighted sum of each column h i in H, which represents the information relevant to the current time step. v t is further integrated with the present state h t to yield the prediction.
Assume a scoring function f : R m ×R m → R which computes the relevance between its input vectors. Formally, we have the following formula to calculate the context vector v t :

Temporal Pattern Attention
While previous work focuses mainly on changing the network architecture of the attention-based models via different settings to improve performance on various tasks, we believe there is a critical defect in applying typical attention mechanisms on RNN for MTS forecasting. The typical attention mechanism selects information relevant to the current time step, and the context vector v t is the weighted sum of the column vectors of previous RNN hidden states, This design lends itself to tasks in which each time step contains a single piece of information, for example, an NLP task in which each time step corresponds to a single word. If there are multiple variables in each time step, it fails to ignore variables which are noisy in terms of forecasting utility. Moreover, since the typical attention mechanism averages the information across multiple time steps, it fails to detect temporal patterns useful for forecasting. The overview of the proposed model is shown in Figure 2. In the proposed approach, given previous RNN hidden states H ∈ R m×(t−1) , the proposed attention mechanism basically attends to its row vectors. The attention weights on rows select those variables that are helpful for forecasting. Since the context vector v t is now the weighted sum of the row vectors containing the information across multiple time steps, it captures temporal information.

Problem Formulation
In MTS forecasting, given an MTS, where ∆ is a fixed horizon with respect to different tasks. We denote the corresponding prediction as y t−1+∆ , and the ground-truth value asŷ t−1+∆ = x t−1+∆ . Moreover, for every task, we use only {x t−w , x t−w+1 , . . . , x t−1 } to predict x t−1+∆ , where w is the window size. This is a common practice [Lai et al.(2018)Lai, Chang, Yang, andLiu, Qin et al.(2017)Qin, Song, Cheng, Cheng, Jiang, andCottrell], because the assumption is that there is no useful information before the window and the input is thus fixed.

Temporal Pattern Detection using CNN
CNN's success lies in no small part to its ability to capture various important signal patterns; as such we use a CNN to enhance the learning ability of the model by applying CNN filters on the row vectors of H. Specifically, we have k filters C i ∈ R 1×T , where T is the maximum length we are paying attention to. If unspecified, we assume T = w. Convolutional operations yield H C ∈ R n×k where H C i,j represents the convolutional value of the i-th row vector and the j-th filter. Formally, this operation is given by

Proposed Attention Mechanism
We calculate v t as a weighted sum of row vectors of H C . Defined below is the scoring function f : R k × R m → R to evaluate relevance: where H C i is the i-th row of H C , and W a ∈ R k×m . The attention weight α i is obtained as Note that we use the sigmoid activation function instead of softmax, as we expect more than one variable to be useful for forecasting. Completing the process, the row vectors of H C are weighted by α i to obtain the context vector Then we integrate v t and h t to yield the final prediction

Analysis of Proposed Attention on Toy Examples
In order to elaborate the failure of traditional attention mechanisms and the influence of interdependencies, we study the performance of different attention mechanisms on two artificially constructed toy examples. In the first toy example, the t-th time step of the i-th time series is defined as sin( 2πit 64 ), that is, each time series is a sine wave with different periods. Notice that any two time series are mutually independent in the first toy example, so there are no interdependency.
The second toy example add interdependencies to the first toy example by mixing time series, and thus the t-th time step of the i-th time series is formulated as: where D is the number of time series. Both toy examples are visualized in Fig. 3 for D = 6.
The two examples with D = 6 are shown in Fig. 4. All models in the following analyses are trained with window size w = 64, horizon ∆ = 1, and similar amount of parameters.

Failure of traditional attention mechanisms
Intuitively, for the first toy example, the model can accurately predict the next value by memorizing the value that appears exactly one period before. However, we know that different time series have different periods, which means to have a good prediction, the model should be able to look back different numbers of time steps for different series. From this point, it is clear that the failure of traditional attention mechanisms comes from extracting only one previous time step while discounting the information in other time steps. On the other hand, our attention mechanism attends on the features extracted from row vectors of RNN hidden states by CNN filters, which enables the model to select relevant information across multiple time steps.
The aforementioned explanation is verified by the left plot in Figure 4, where we observe that the performance of the LSTM with Luong attention is poor when D 1, compared to the others. Notice that all models have similar amount of parameters, which implies that the LSTM without attention has a larger hidden size when compared to the LSTM with Luong attention. Consequently, the LSTM without attention outperforms the LSTM with Luong attention when D 1, because the larger hidden size helps the model to make prediction while the Luong attention is nearly useless. On the contrary, our attention is useful, so the LSTM with our attention is better than the LSTM without attention on average, even though its hidden size is smaller.

Influence of interdependencies
When there are interdependencies in MTS data, it is desirable to leverage the interdependencies to further improve forecasting accuracy. The right plot in Figure 4 shows that both the LSTM with Luong attention and the LSTM without attention do not benefit from the added interdependencies, since the loss values remain the same. On the other hand, the loss of the LSTM with the proposed attention is lower when there are interdependencies, which suggests that our attention successfully utilized the interdependencies to facilitate MTS forecasting.

Experiments and Analysis
In this section, we first describe the datasets upon which we conducted our experiments. Next, we present our experimental results and a visualization of the prediction against LSTNet. Then, we discuss the ablation study. Finally, we analyze in what sense the CNN filters resemble the bases in DFT.

Datasets
To evaluate the effectiveness and generalization ability of the proposed attention mechanism, we used two dissimilar types of datasets: typical MTS datasets and polyphonic music datasets.
The These datasets are real-world data that contains both linear and non-linear interdependencies. Moreover, the Solar Energy, Traffic, and Electricity datasets exhibit strong periodic patterns indicating daily or weekly human activities.
According to the authors of LSTNet, all datasets have been split into training (60%), validation (20%), and testing set (20%) in chronological order. In contrast, the polyphonic music datasets introduced below are much complicated, in the sense that no apparent linearity or repetitive patterns exist: -MuseData [Nicolas Boulanger-Lewandowski and Vincent(2012)]: a collection of musical pieces from various classical music composers in MIDI format. -LPD-5-Cleansed [Hao-Wen Dong and Yang(2018), Raffel(2016)]: 21, 425 multi-track piano-rolls that contain drums, piano, guitar, bass, and strings.
To train models on these datasets, we consider each played note as 1 and 0 otherwise (i.e., a musical rest), and set one beat as one time step as shown in Table 1. Given the played notes of 4 bars consisting of 16 beats, the task is to predict whether each pitch at the next time step is played or not. For training, validation, and testing sets, we follow the original MuseData separation, which is divided into 524 training pieces, 135 validation pieces, and 124 testing pieces. LPD-5-Cleansed, however, was not split in previous work [Hao-Wen Dong and Yang(2018), Raffel(2016)]; thus we randomly split it into training (80%), validation (10%), and testing (10%) sets. The size of LPD-5-Cleansed dataset is much larger than others, so we decided to use a smaller validation and testing set. The statistics of both the typical MTS datasets and polyphonic music datasets are summarized in Table 1.

Methods for Comparison
We compared the proposed model with the following methods on the typical MTS datasets: -AR: standard autoregression model. However, as both traditional baseline methods and LSTNet are ill-suited to polyphonic music datasets due to their non-linearity and the lack of periodicity, we use LSTM and LSTM with Luong attention as the baseline models to evaluate the proposed model on polyphonic music datasets: -LSTM: RNN cells as introduced in Section 3.

Model Setup and Parameter Settings
For all experiments, we used LSTM units in our RNN models, and fixed the number of CNN filters at 32. Also, inspired by LSTNet, we included an autoregression component in our model when training and testing on typical MTS datasets.
For typical MTS datasets, we conducted a grid search over tunable parameters as done with LSTNet. Specifically, on Solar Energy, Traffic, and Electricity, the range for window size w was {24, 48, 96, 120, 144, 168}, the range for the number of hidden units m was {25, 45, 70}, and the range for the step of the exponential learning rate decay with a rate of 0.995 was {200, 300, 500, 1000}. On Exchange Rate, these three parameters were fixed at 30, 6 and 120, respectively. Two types of data normalization were also viewed as part of the grid search: one normalized each time series by the maximum value in itself, and the other normalized every time series by the maximum value over the whole data set. Lastly, we used the absolute loss function and Adam with a 10 −3 learning rate on Solar Energy, Traffic, and Electricity, and a 3 · 10 −3 learning rate on Exchange Rate. For the other compared methods as mentioned in previous subsection, the parameters were identical to the numbers reported in the LSTNet paper [Lai et al.(2018)Lai, Chang, Yang, and Liu].
For models used for the polyphonic music datasets, including the baselines and proposed models in the following subsections, we used 3 layers for all RNNs, as done in [Chuan and Herremans(2018)], and fixed the trainable parameters to around 5 · 10 6 by adjusting the number of LSTM units to fairly compare different models. In addition, we used the Adam optimizer with a 10 −5 learning rate and a cross entropy loss function.

Evaluation Metrics
On typical MTS datasets, since we compared the proposed model with LST-Net, we followed the same evaluation metrics: RAE, RSE and CORR. The first metric is the relative absolute error (RAE), which is defined as The next metric is the root relative squared error (RSE):  Table 2 Results on typical MTS datasets using RAE, RSE and CORR as metrics. Best performance in boldface; second best performance is underlined. We report the mean and standard deviation of our model in ten runs. All numbers besides the results of our model is referenced from the paper of LSTNet [Lai et al.(2018)Lai, Chang, Yang, and Liu].  and finally the third metric is the empirical correlation coefficient (CORR): where y,ŷ is defined in Section 4.1,ŷ t , ∀t ∈ [t 0 , t 1 ] is the label of the testing data, and y denotes the mean of set y. RAE and RSE both disregards data scale and is a normalized version of the mean absolute error (MAE) and the root mean square error (RMSE), respectively. For RAE and RSE, the lower the better, whereas for CORR, the higher the better. To decide which model is better on polyphonic music datasets, we use validation loss (negative log-likelihood), precision, recall, and F1 score as measurements which are widely used in work on polyphonic music generation [Nicolas Boulanger-Lewandowski and Vincent(2012), Chuan and Herremans(2018)].

Results on Typical MTS Datasets
On typical MTS datasets, we chose the best model on the validation set using RAE/RSE/CORR as the metric for the testing set. The numerical results are tabulated in Table 2, where the metric of the first two tables are RAE, followed by two tables of RSE metric, and ended by another two tables using CORR metric. Both tables show that the proposed model outperforms all  Table 3 Precision, recall, and F1 score of different models on polyphonic music datasets other methods on all datasets, horizons, and metrics, with a single exception. Generally speaking, the larger the D in Table 1, the better our model performs. Also, our models are able to deal with a wide range of dataset size, from the smallest 534 KB Exchange Rate dataset to the largest 172 MB Solar Energy dataset. In these results, the proposed model consistently demonstrates its superiority for MTS forecasting.
In the comparison to LSTNet-Skip and LSTNet-Attn, the previous state-ofthe-art methods, the proposed model exhibits superior performance, especially on Traffic and Electricity, which contain the largest amount of time series. Moreover, on Exchange Rate, where no repetitive pattern exists, the proposed model is still the best overall; the performance of LSTNet-Skip and LSTNet-Attn fall behind traditional methods, including AR, LRidge, LSVR, and GP. The proposed model is outperformed by LRidge on Exchange Rate with the 6-day horizon, because linear models is sufficient for this dataset, and deep learning is redundant. In Figure 5 we also visualize and compare the prediction of the proposed model and LSTNet-Skip.
In summary, the proposed model achieves state-of-the-art performance on both periodic and non-periodic MTS datasets.

Results on Polyphonic Music Datasets
In this subsection, to further verify the efficacy and generalization ability of the proposed model, we describe experiments conducted on polyphonic music datasets; the results are shown in Figure 6 and Table 3. We compared three RNN models: LSTM, LSTM with Luong attention, and LSTM with the proposed attention mechanism. Figure 6 shows the validation loss across training epochs, and in Table 3, we use the models with the lowest validation loss to calculate precision, recall, and F1 score on the testing set.
From the results, we first verify our claim that the typical attention mechanism does not work on such tasks, as under similar hyperparameters and trainable weights, LSTM and the proposed model outperform such attention mechanisms. In addition, the proposed model also learns more effectively com- Fig. 7 Magnitude comparison of (1) DFT of CNN filters trained on Traffic with a 3-hour horizon, and (2) every window of the Traffic dataset. To make the figure more intuitive, the unit of the horizontal axis is the period. pared to LSTM throughout the learning process and yields better performance in terms of precision, recall, and F1 score.

Analysis of CNN Filters
DFT is a variant of the Fourier transform (FT) which handles equally-spaced samples of a signal in time. In the field of time series analysis, there is a wide body of work that utilizes FT or DFT to reveal important characteristics in time series [N.E. Huang andLiu(1998), Bloomfield(1976)]. In our case, since the MTS data is also equally-spaced and discrete, we could apply DFT to analyze it. However, in MTS data, there is more than one time series, so we naturally average the magnitude of the frequency components of every time series, and arrive at a single frequency domain representation. We denote this the average discrete Fourier transform (avg-DFT). The single frequencydomain representation reveals the prevailing frequency components of the MTS data. For instance, it is reasonable to assume a notable 24-hour oscillation in Figure 5, which is verified by the avg-DFT of the Traffic dataset shown in Figure 7.  Since we expect our CNN filters to learn temporal MTS patterns, the prevailing frequency components in the average CNN filters should be similar to that of the training MTS data. Hence, we also apply avg-DFT on the k = 32 CNN filters that are trained on Traffic with a 3-hour horizon; in Figure 7 we plot the result alongside with the avg-DFT of every window of Traffic dataset. Impressively, the two curves reach peaks at the same periods most of the time, which implies that the learned CNN filters resemble bases in DFT. At the 24, 12, 8, and 6-hour periods, not only is the magnitude of the Traffic dataset at its peak, but the magnitude of CNN filters also tops out. Moreover, in Figure 8, we show that different CNN filters behave differently. Some specialize at capturing long-term (24-hour) temporal patterns, while others are good at recognizing short-term (8-hour) temporal patterns. As a whole, we suggest that the proposed CNN filters play the role of bases in DFT. As demonstrated in the work by [Rippel et al.(2015)Rippel, Snoek, and Adams], such a "frequency domain" serves as a powerful representation for CNN to use in training and modeling. Thus, LSTM relies on the frequency-domain information extracted by the proposed attention mechanism to accurately forecast the future.

Ablation Study
In order to verify that the above improvement comes from each added component rather than a specific set of hyperparameters, we conducted an ablation study on the Solar Energy, Traffic, Electricity, and MuseData datasets. There were two main settings: one controlling how we attend to hidden states, H, of RNN and the other controlling how we integrate the scoring function f into the proposed model, or even disable the function. First, in the proposed method, we let the model attend to values of various filters on each position (H C i ); we can also consider attending to values of the same filters at various positions ((H C ) i ) or row vectors of H (H i ). These three different approaches correspond to the column headers in Table 4: "Position", "Filter", and "Without CNN". Second, whereas in the typical attention mechanism, softmax is usually used on the output value of scoring function f to extract the most relevant information, we use sigmoid as our activation function. Therefore, we compare these two different functions. Another possible structure for forecasting is to concatenate all previous hidden states and let the model automatically learn which values are important. Taking these two groups of settings into consideration, we trained models with all combinations of possible structures on these four datasets.
The MuseData results show that the model with sigmoid activation and attention on H C i (position) is clearly the best, which suggests that the proposed model is reasonably effective for forecasting. No matter which proposed component is removed from the model, performance drops. For example, using softmax instead of sigmoid raises the negative log-likelihood from 0.04878 to 0.04931; we obtain a even worse model with a negative log-likelihood of 0.4987 if we do not use CNN filters. In addition, we note no significant improvement between the proposed model and that model using softmax on the first three datasets in Table 4: Solar Energy, Traffic, and Electricity. This is not surprising, given our motivation for using sigmoid, as explained in Section 4.3. Originally, we expected CNN filters to find basic patterns and expected the sigmoid function to help the model to combine these patterns into one that helps. However, due to the strongly periodic nature of these three datasets, it is possible that using a small number of basic patterns is sufficient for good prediction. Overall, however, the proposed model is more general and yields stable and competitive results across different datasets.

Conclusions
In this paper, we focus on MTS forecasting and propose a novel temporal pattern attention mechanism which removes the limitation of typical attention mechanisms on such tasks. We allow the attention dimension to be featurewise in order for the model learn interdependencies among multiple variables not only within the same time step but also across all previous times and series. Our experiments on both toy examples and real-world datasets strongly support this idea and show that the proposed model achieves state-of-the-art results. In addition, the visualization of filters also verifies our motivation in a more understandable way to human beings.