{'fraction_non_alphanumeric': 0.03120476113881293, 'fraction_numerical': 0.009650957053241113, 'mean_word_length': 4.9897238278741165, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
RTFN: Robust Temporal Feature Network

Time series analysis plays a vital role in various applications, for instance, healthcare, weather prediction, disaster forecast, etc. However, to obtain sufficient shapelets by a feature network is still challenging. To this end, we propose a novel robust temporal feature network (RTFN) that contains temporal feature networks and attentional LSTM networks. The temporal feature networks are built to extract basic features from input data while the attentional LSTM networks are devised to capture complicated shapelets and relationships to enrich features. In experiments, we embed RTFN into supervised structure as a feature extraction network and into unsupervised clustering as an encoder, respectively. The results show that the RTFN-based supervised structure is a winner of 40 out of 85 datasets and the RTFN-based unsupervised clustering performs the best on 4 out of 11 datasets in the UCR2018 archive.


Introduction
In the real world, time series data are collected from various domains, such as weather forecast, human heart record and animal trajectory [1]. How to make full use of the data in real-world applications is crucial, which depends on how well features are extracted. Recently, sufficient feature extraction has become a critical challenge, which is also a basis for time series classification [2].
In the past two decades, there are primarily two solutions to address the challenge above, including traditional approaches and deep learning algorithms [3]. Compared with the former, deep learning has been reported to achieve better performance in feature extraction. In particular, attention networks usually achieve excellent performance in time series classification, because attention mechanism can relate different positions of a sequence in order to derive sensitive representations and relationships [4]. Besides, to embed long short term memory network (LSTM) into temporal networks is able to explore long-and short-period connections and relationships among data, enhancing feature extraction and thus facilitating the processing of subsequent tasks [5] [6] [7]. In short, attention and LSTM have been regarded as promising structures in the area of time series processing.
To facilitate sufficient feature extraction, this paper proposes a novel robust temporal feature network (RTFN), which consists of temporal feature networks and attentional LSTM networks. The temporal feature networks are based on residual networks [8] and multi-head convolution neural networks, which are responsible for extracting basic temporal features. In each attentional LSTM network, LSTM is embedded into an attentional structure for the purpose of efficiently mining the hidden shapelets and relationships that are usually ignored by the temporal networks. In the experiments, we embed RTFN into a supervised structure, and test it on 85 standard datasets in the UCR2018 archive, where the RTFN-based classification algorithm outperforms a number of state-of-the-art supervised algorithms on 40 datasets. Moreover, we also use RTFN in a simple unsupervised representation clustering as an encoder. The RTFN-based unsupervised clustering is a winner of 4 out of 11 datasets in the UCR2018 archive, compared with other unsupervised temporal clustering algorithms.
The rest of the paper is organized as follows. Section 2 reviews the attention and LSTM techniques. Section 3 introduces the overview of RTFN, the Attentional LSTM, the RTFN-based supervised structure and the RTFN-based unsupervised clustering. Experimental results and analysis are given in Section 4. In Section 5, summary and conclusions are provided.

Related Work
Feature extraction is essential for many tasks, such as computer vision, nature language processing, reasoning, etc. In general, robust feature networks are composed of targeted feature meta-networks, which take their own advantages to capture the existing features from domain data and explore the relationships among them. Attention and LSTM are two types of well recognized meta-networks in recent years, promoting the rapid development in a variety of application domains.

Attention
The attention mechanism aims at mapping a given query and a set of key-value pairs to an output, where the output is a weighted sum of the values. For each value, its associated weight is calculated based on a compatibility function of the query with the corresponding key [10]. For any input data, attention can mine different positions to derive the representation of input data.
Just like our eyes, the attention mechanism focuses on the sensitive information we look for, capture the areas we are interested in, and discover rich features and relationships among them. Its use is to discover the hidden information and relationships that classical networks are unaware of, which helps to further complement the time-series and semantic features. Attention has been applied successfully in various areas, such as object detection [11], image caption [12], activity recognition [13], speech recognition [14], question answering [15] and software-defined networking [16].

LSTM
LSTM is a widely used recurrent neural network. For a sequential model, LSTM can explore the features of input sequence by using specific internal units, and make use of these features to mine more underlying information and relationships from the internal structure [17]. Besides, the internal memory units in LSTM store the new features transmitted to the next processing units. In this way, when the features being transmitted are forgotten, they can be recovered by those stored in the internal memory units. Hence, LSTM can obtain abundant features from input data. Recently, LSTM has been incorporated into various network structures for addressing time-series tasks, for instance, ERA-LSTM [5], FCN-LSTM [6] and multivariate FCN-LSTM [18].
Motivated by the research related to the attention mechanism and LSTM, we design a robust temporal feature network that integrates LSTM into an attentional learning framework, which is named as attentional LSTM network. By taking advantages of the two techniques, the proposed network is able to not only explore long and short term temporal features, but also focus on different positions of a sequence to capture the shapelets hiding in the input data.

RTFN
This section first overviews the structure of RTFN, and then describes the attentional LSTM network in detail. In the end, the RTFN-based supervised structure and unsupervised clustering are introduced.

Overview
The structure of RTFN is shown in Fig. 1. It primarily consists of temporal feature networks and attentional LSTM networks. The former is based on residual networks and multi-head convolution neural networks, responsible for capturing basic temporal features from the input time-series data. Especially, in the temporal feature networks, we place self-attention networks [10] between the two multi-head convolutional neural networks to relate various positions of the shapelets that the first multi-head convolutional neural networks obtain, which enriches temporal features. The latter pays attention to digging out the complex temporal relationships from the input, which helps to compensate for the loss of features that the temporal feature networks may not capture. Therefore, combining the two types of networks together promotes the abundance of both the basic and in-depth temporal features. In this paper, RTFN is embedded into supervised structure first and unsupervised clustering later, in the context of time series classification.

Attentional LSTM
Motivated by the original attention mechanism, we propose an innovative Attentional LSTM to concentrate on mutual temporal relationships. Its details are shown in Fig. 2. An attentional LSTM network is a hybridized network that incorporates LSTM networks into an attentional structure. In this network, a temporal query and a set of key-value pairs are mapped to an output, where the query, key, and value are vectors obtained from the feature extraction by the LSTM networks. The output is defined as a weighted sum of the values with sufficient shapelets and relationships. In this paper, each value is obtained by a compatibility function that helps to mine the hidden relationships between a query and its corresponding key that already carry basic features, thus strengthening the robustness of the attentional LSTM network. The calculation of the attentional LSTM network is defined in Eq. (1).
where f Q , f K and f V represent the matrixes of the queries, keys and values, respectively.
After the sequential data go through the LSTM networks, the queries, keys and values carry sufficient long-and short-term features as input, which are then used to capture the in-depth shapelets and relationships hiding behind as output.

The RTFN-based Supervised Structure
The RTFN-based supervised structure is shown in Fig. 3, where a dropout layer and a fully-connected layer are cascaded to the output of RTFN. To be specific, we introduce the dropout layer to avoid overfitting during the training process. The fully-connected layer performs as the classifier of the RTFN-based supervised structure. The reason we simply use the dropout and fully-connected layers is that the features extracted by RTFN are sufficiently good and thus a complicated classifier network is not necessary. Like other commonly used supervised algorithms, we use the cross entropy function to compute the difference between the prediction result and ground truth label, shown as Eq. (2).
where Y train i represents the ground truth label of the i-th time-series class, and p i {i = 1, 2, ..., n} is the prediction output of the RTFN-based supervised structure.

The RTFN-based Unsupervised Clustering
The RTFN-based unsupervised clustering is based on a widely adopted auto-encoder unsupervised architecture, as shown in Fig. 4. To be specific, it is primarily composed of RTFN as the encoder, a decoder and a K-means algorithm [19]. RTFN is responsible for obtaining basic and in-depth shapelets from the input data as many as possible. The decoder is made up of four fully-connected layers, helping to reconstruct the features captured by RTFN. Besides, the K-means algorithm acts as unsupervised classifier.
Different from that the total loss takes k-means loss into account in [20][21] [22], the loss of the RTFN-based unsupervised clustering simply depends on a reconstruction loss defined in Eq. (3).
whereX train i is the input data and X rec i represents the decoder output.

Experiments
This section first introduces the experimental setup, and then evaluates the RTFN-based supervised structure and unsupervised clustering, respectively.

Experimental Setup
The UCR 2018 archive is one of the authoritative temporal data archives, which contains 128 datasets with different lengths in a variety of application areas. We select 85 and 11 standard datasets to evaluate performance of the RTFN-based supervised structure and unsupervised clustering, respectively. All experiments are run on a computer with Ubuntu 18.04 OS, Nvidia dual-GTX 1070Ti GPU with 8GB × 2 GRAM and AMD R5 1400 CPU with 16G RAM. For encouraging scientific comparison, our code is available at http://X.X.X.X.

Evaluation of the RTFN-based Supervised Structure
To evaluate performance of the RTFN-based supervised structure, we compare it with a number of state-of-the-art supervised algorithms against the top-1 accuracy. We use 'win', 'tie' and 'lose' to rank the algorithms for comparison, i.e. on how many datasets that an algorithm performs better than, equivalent to, or worse than the others. For each algorithm, the number of the 'best' cases is the summation of the numbers of the corresponding 'win' and 'tie' cases. Table 1 shows the statistical results obtained by different algorithms on the 85 selected datasets in the UCR 2018 archive. For each dataset, the existing SOTA represents the best algorithm on that dataset [4], including BOSS [23], COTE [24], DTW [25] and so on; similarly, for each dataset, the Best:lstm-fcn is the best-performance approach on that dataset, e.g. it involves LSTM-FCN, ALSTM-FCN, etc, in [6]. Note that the existing SOTA did not consider the last 20 of the 85 datasets.
The original results are shown in the appendix. It is seen that the proposed RTFN-based supervised structure performs the best in 'tie' and the second best in 'win', guaranteeing its first position in 'best'. To be specific, ours wins in 11 cases and performs no worse than any other algorithm in 29 cases, which leads to 40 'best' cases in the competition. Besides, the Best: lstm-fcn and the Vanilla: ResNet-Transormer achieve the second and third 'best' performance, with respect to the 'best' score. Especially, the former is a winner on 16 datasets, indicating its outstanding performance. On the other hand, the ResNet-Transformer 2 [4] is the worst algorithm, with 2 'win' scores and 20 'tie' scores only.
In addition, to further reflect the difference between the proposed structure and the others in the long time-series datasets, we select 12 out of the 85 datasets (see Table 2 for details) and show the top-1 accuracy results in Table 3. Besides, we add the RTFN-based supervised structure without the attentional LSTM (ours w/o a-LSTM) to the comparison, which helps to verify the contribution of the attentional LSTM.
On the one hand, if focusing on ours and ours w/o a-LSTM, one can easily see the former outperforms the latter on each dataset, as the former always obtains higher accuracy. This clearly unveils the attentional LSTM plays a non-trivial role in the performance improvement, helping our structure to gain 8 'best' scores and the highest mean accuracy. This is because the embedded attentional LSTM is capable of extracting the underlying temporal shapelets and relationships that are simply ignored by the temporal feature networks.

Evaluation of the RTFN-based Unsupervised Clustering
To evaluate performance of the RTFN-based unsupervised clustering, we compare it with a number of state-of-the-art unsupervised algorithms against three metrics, including the 'best' score as used in Section 4.2, the average ranking, and the rand index [26]. The average (AVG) rank scores are computed according to the average Geo-ranking approach, which measures the average difference between the accuracies of a model and the best accuracies among all models [22]. As a widely adopted performance indicator, the rand index (RI), RI, is defined in Eq. (4).

RI = P T P + N T P s(s − 1)/2 (4)
where P T P and N T P are the numbers of the positive and negative time-series pairs in the clustering, respectively, and s is the dataset size.  [36] and DTCR [22] both achieve 4 'best' scores, which are, to some extent, statistically equivalent to ours. This is because the pseudo-labels help to guide the training process in USSL while the mistake correction helps to facilitate the temporal reconstruction in DTCR. However, for USSL, there is no efficient mechanism to alleviate the negative impact on the training process when mistakes exist in pseudo-labels. Besides, the temporal reconstruction in DTCR is not good at mining cluster-specific representations from input data. This is why DTCR needs an additional K-means objective to its seq2seq structure. On the contrary, our unsupervised clustering simply adopts an auto-encoding structure to update the parameters of our model and utilizes a K-means algorithm to classify the features obtained by RTFN.
Although it is quite simple in structure, ours achieves decent performance on the 11 datasets, which depends on the strong feature extraction ability of RTFN.
In order to further testify that the proposed networks contribute to the accuracy improvement of the K-means algorithm, we compare the RTFN-based unsupervised clustering with a separate K-means algorithm on the 11 datasets above and show the RI results in Fig. 5. One can see that ours outperforms the separate K-means algorithm on each dataset. Sometimes, RI can be significantly improved, such as, on Wafer, Prox.Phal.out.ageGr and OSULeaf datasets. This is because our unsupervised clustering is provided with sufficient features obtained by the proposed RTFN, especially those hiding deeply in the input data which are beyond the exploration abilities of ordinary feature extraction networks. The results of the AVG ranks of all algorithms for comparison are shown in Fig. 6. Figure 6: Results of the AVG ranks of the 15 algorithms.

Conclusions
In the proposed RTFN, the temporal feature networks are responsible for extracting basic features while the attentional LSTM networks target on those hidden shapelets and relationships which are extremely difficult to mine. Experimental results demonstrate that our RTFN achieves decent performance in both supervised classification and unsupervised clustering. Specifically, our RTFNbased supervised algorithm performs the best on 40 datasets, compared with the latest results from the supervised classification community. In particular, ours wins on 8 out of 12 long time-series datasets. Our RTFN-based unsupervised algorithm overweighs 14 state-of-the-art unsupervised algorithms on 11 datasets. Last but not least, the experimental results also indicate that RTFN has good potential to be embedded into other learning frameworks to handle time-series problems of various domains in the real world.

Broader Impact
RTFN is a feature extraction network used to mine sufficient features from raw data. The features extracted by RTFN are beneficial to both supervised classification and unsupervised clustering, especially in time series classification. The high accuracy brought by RTFN facilitates development of useful applications in our daily life. For example, in the medical health domain, accurate ECG recognition improves clinical work efficiency of doctors and health professionals, promoting the development of AI medical care. In the earthquake warning domain, accurate classification is of great help for earthquake prediction, which saves human lives and reduces property losses. In the wildlife conservation domain, precise trajectory prediction also helps us to observe activity and behavior patterns of wild animals, which is in favor of their protection. On the other hand, due to its high complexity, RTFN cannot be directly deployed on mobile devices now, such as, mobile phones or intelligent watches. In addition, we do not think RTFN brings any negative impact in terms of its ethical aspects and future societal consequences.