{'fraction_non_alphanumeric': 0.03747085915733212, 'fraction_numerical': 0.012736680502643998, 'mean_word_length': 4.534298300818125, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Short-term traffic flow forecasting with spatial-temporal correlation in a hybrid deep learning framework

Deep learning approaches have reached a celebrity status in artificial intelligence field, its success have mostly relied on Convolutional Networks (CNN) and Recurrent Networks. By exploiting fundamental spatial properties of images and videos, the CNN always achieves dominant performance on visual tasks. And the Recurrent Networks (RNN) especially long short-term memory methods (LSTM) can successfully characterize the temporal correlation, thus exhibits superior capability for time series tasks. Traffic flow data have plentiful characteristics on both time and space domain. However, applications of CNN and LSTM approaches on traffic flow are limited. In this paper, we propose a novel deep architecture combined CNN and LSTM to forecast future traffic flow (CLTFP). An 1-dimension CNN is exploited to capture spatial features of traffic flow, and two LSTMs are utilized to mine the short-term variability and periodicities of traffic flow. Given those meaningful features, the feature-level fusion is performed to achieve short-term forecasting. The proposed CLTFP is compared with other popular forecasting methods on an open datasets. Experimental results indicate that the CLTFP has considerable advantages in traffic flow forecasting. in additional, the proposed CLTFP is analyzed from the view of Granger Causality, and several interesting properties of CLTFP are discovered and discussed .


Introduction
The accurate and reliable forecasting of short-term traffic flow is the precursor of a multitude of intelligent transportation systems (ITS) such as proactive dynamic traffic control, intelligent route guidance and intelligent location-based service, thus it is always a hot topic in ITS. Recent neural network (CNN) (Krizhevsky, Sutskever, and Hinton, 2012), it uses filters to find relationships between neighboring inputs, which can make it easier for the network to converge on the correct solution. And one of the most successful architecture to characterize long-term memory is long short-term memory network (LSTM) (Graves, Jaitly, and Mohamed, 2013), which learns both short-term and long-term memory by enforcing constant error flow through the designed cell state (Hochreiter and Schmidhuber, 1997).
Motivated by the success of CNN and LSTM, and with consideration of the spatial-temporal characteristics of traffic flow, we propose a novel short-term traffic flow prediction method based on the combination of CNN and LSTM (CLTFP). A deep convolution neural network is utilized to mine the space features of traffic flow data. LSTMs are employed to learn features of both short-term time variation and long-term periodicity. Then we feed those spatial-temporal features into a linear regression layer to predict future traffic flow. In feature-level based data fusion, it is natural to assume that a small portion of features have strong impact (Bishop, 2006). Thus in order to strengthen the sparsity of features, we add a l 1 norm constraint on weights of the regression layer. Finally, we train the neural network end-to-end. Our method is evaluated on traffic flow of a freeway corridor collected from an open datasets, the results show that our method exhibits better performance than state-of-arts. Moreover, we analyze the features captured by CLTFP in terms of incremental predictability, the analysis graphically demonstrate how black-box typed CLTFP understand causality between future-past traffic flow.

Related works
The deep architecture for short-term traffic flow forecasting has recently been studied in ITS, but mainly on deep fully-connected architecture. Huang et al. (2014) used a deep belief network to capture the spatial-temporal features of traffic flow and proposed a multi-task learning architecture to perform exit station flow and road flow forecasting. Similarly, Lv et al. (2015) proposed a stacked autoencoder model based short-term traffic flow forecasting. Tan et al. (2016b) investigated the impact of pre-traning with different deep belief networks on the DNN based short-term traffic flow forecasting. Chen et al. (2016) developed a Stack denoise Autoencoder to learn hierarchical representation of urban traffic flow. Polson and Sokolov (2016) used a deep neural network architecture to forecast traffic flows during special events. The above approaches have been able to accurately forecast future traffic flow to some degree, but they did not exploit the topological locality and long-term memory of traffic flow, which hindered their predictive power. Motivated by the success of LSTM, Ma et al. (2015) applied LSTM to short-term traffic forecasting, they claimed that LSTM can capture long-term memory of traffic flow. However, despite the efficient usage of long temporal dependency, the spatial dependency is not fully utilized in their work.
Several attempts have been made to combine CNN and LSTM architectures especially for connection of computer vision and natural language processing. For example, several methods have made use of CNN features and LSTM for image/video description generation (Vinyals et al., 2015;Yao et al., 2015;Peris et al., 2016). The combination of CNN and LSTM have also been successfully applied to visual activity recognition (Donahue et al., 2015), sentiment analysis  and video classification (Wu et al., 2015). As traffic flow data share some common properties with visual data (Yang, Kalpakis, and Biem, 2014) and language, the success of combining CNN and LSTM on computer vision and language processing indicates the potential of such combination on traffic flow forecasting.

Model
In this section, we describe our CNN-LSTM based short-term traffic flow prediction method (CLTFP).
See Fig. 1 for the graphical illustration of the proposed model. It can be found from Fig.1 that CLTFP consists of a 1D CNN, two LSTM RNNs and a fully-connected layer, we will describe each part in the following subsection. Figure 1: CLTFP, our model, is based end-to-end on a neural network consisting of a 1D CNN (capture spatial features), two LSTM RNNs (capture short-term and periodic features) and followed by a fully connected layer to fuse all features to forecast traffic flow at target time point t.

Spatial features captured by CNN
Suppose we need to forecast traffic flow of p locations {s i } p i=1 in (t, t + 1, · · · , t + h), in which h is the prediction horizon. The historical traffic flow data of p locations {s i } p i=1 in (t − n, t − 1) are traditionally used as inputs for generating prediction in (t, t + 1, · · · , t + h). If we put the historical data together, we can get a data matrix The traffic flow usually depends on traffic flow of that location and its neighbors. As Convolutional Neural Networks (CNNs) have been successful in handling data representation with a locality structure, we naturally adapt a 1-dimensional CNN to capture spatial features of traffic flow. Our 1D CNN does not attack time mode. Instead, the time dimension is treated as channels of an image, which means that we only perform convolution on vectors For locations of a freeway corridor given in Fig.1, the traffic travel through from upstream s 1 to downstream s p , the conventional 1D CNN is naturally exploited to capture spatial features of such a transportation network, where the k-th feature map is obtained as follows where w k q is the weights vector, b k q is the bias, o c denotes a nonlinear activation and * denotes the convolution. The pooling layers are not employed in our model. Because it is evident that an all convolution net achieves better performance on small images recognition (Springenberg et al., 2014), and the space dimension of traffic data on short-term traffic forecasting task is always limited.
There are many more complex transportation networks than the one given in Fig.1, e.g. a transportation network of a big city. In these cases, the conventional 1D CNN can not be used without any modifications. However, the traffic flow in any transportation network always has some graph structures (Shahsavari and Abbeel, 2015), Thus the CNN on graph-structured data proposed by Henaff, Bruna, and LeCun (2015) can be an alternative for such transportation networks.

Short-term temporal features
As stated in Ma et al. (2015), the traditional forecasting models mainly suffer from two drawbacks in handling time mode information of traffic flow: (1) Traditional methods especially traditional RNNs are difficult to train if the traffic flow series has long time lags, which means traditional RNNs will provide poor performance if the time window size n in Eq.(1) is too large. (2) It is difficult to find the optimal time window size n, as the correlation between traffic flow of different time points is affected by many complex factors such as weather, speed and unpredictable incident. The LSTM is one of the more practical ways to tackle these issues, thus we propose to use LSTM to capture the time mode information of traffic flow. Different from the work of Ma et al. (2015), which directly used LSTM to generate predictions of traffic flow, we use LSTM to generate time features at each time point and build a deeper and more complex traffic flow forecasting model. Similar to the traditional RNNs, a LSTM structure is composed of one input layer, one or several hidden layers and one output layer. The core idea of LSTM is the memory cell in hidden layers, which is designed to avoid the gradient vanish and explosion in traditional RNNs. As shown in Fig.   2, a memory cell contains four main parts: an input gate, a neuron with a self-recurrent connection, a forget gate and an output gate. For the generation of short-term temporal features, the inputs of LSTM is denoted as T = (T 0 , T 1 , · · · , T n−1 ) where T q = s 1 (t − n + q), s 2 (t − n + q), · · · , s p (t − n + q) T in Eq (1), and the output temporal features in each historical time point is denoted as H = (H 0 , H 1 , · · · , H n−1 ), n is the time window size. The generated temporal features are iteratively calculated by following equations: where · represents the Hadamard product of two vectors, and σ(.) and σ h (.) are activation functions.
σ(.) is traditionally set to be a function of domain [0, 1] to control the information flow through time. And σ h (.) is often set to be a centered activation function.

Periodic features
People are used to repeating some behaviors on a same time period of day, e,g. people routinely go to work in the morning and go home in the evening, This is why we can observe a strong periodicity within traffic flow. The periodicity of traffic flow have been identified as a major contributing factor for the traffic flow forecasting. A desirable model that successfully characterize the periodicities can accurately forecast future traffic flow. In this subsection, we propose to use LSTM to capture the features of periodicities for traffic flow forecasting. The inputs for periodicities at time point t are given as following: where t d and t w denote the same time point of t in last day and last weekday respectively, n d and n w denote the time lags of daily periodicity and weekly periodicity respectively.
For features of daily periodicity, the input of LSTM is denoted as (8). The same input can be easily adapted to weekly periodicity. It is obvious that there exists correlation between S d and S d , so as illustrated in Fig.1, the connection between LSTMs for daily periodicity and weekly periodicity are added. With the LSTM architecture given in Fig.1

Feature-level fusion
As shown in Fig. 1, the proposed CLTFP captures spatial features h 1 , h 2 , · · · , h nc , short-term temporal features H 0 , H 1 , · · · , H n−1 , weekly periodic features H w 0 , H w 1 , · · · , H w 2n w , and daily periodic features H d 0 , H d 1 , · · · , H d 2n d . In order to fuse them to perform short-term traffic flow forecasting, we concatenate all the features sequentially into a feature vector, then add a regression layer to perform forecasting. The objective function of regression is the sum of square errors between predicted value s 1 (t), s 2 (t), · · · , s n (t) and future value s 1 (t), s 2 (t), · · · , s n (t). There may be redundancies between the features captured by CLTFP. To handle the feature redundancy problem, we add a sparsity regularization in weights of fully connection layer thus our model is likely to assign a weight close to zero to redundant feature.

Experiments
In this section, we use traffic flow data from PeMS(http://pems.eecs.berkeley.edu/) to evaluate the proposed CLTFM. The CLTFM is compared with several state-of-the-art forecasting methods with deep architecture: LSTM (Ma et al., 2015), SAE (Lv et al., 2015), a shallow neural network and the gradient boosting regression tree (GBRT) method (Zhang and Haghani, 2015). In additional, we perform a study to discover the performance contribution of different features of CLTFM from the view of Granger Causality. All experiments are performed by a PC (CPU: Intel Xeon(R) E5-2620 2.1GHz, 64GB memory, GPU: NVIDIA Tesla K40C).

Datasets
The

Experimental setup
For all methods, time window size n of S is set as 15, which means that 75 min historical data are used to perform forecasting of next 5 min. The time lags of daily periodicity n d and weekly periodicity n w for long-term inputs of CLTFP are set as 6, which means that the traffic flow before and after 30 minutes in previous day and weekday are used to generate forecasting.
For 1D CNN structure of CLTFP, a 3-layer fully convolution structure is used, there are 30 filters in each layer, the filter lengths of first 2 layers are set as 3, the filter length of last layer is set as 2, SReLu (Jin et al., 2015) is used as the activation function of CNN. For the LSTM capturing short-term temporal features H 0 , H 1 , · · · , H n−1 , the feature dimension of each time point is set as 40. For the LSTM capturing long-term features H d 0 , H d 1 , · · · , H d 2n d and H w 0 , H w 1 , · · · , H w 2n w , the feature dimension of each time point is set as 25. For the regression layer of CLTFP, the l 1 norm regularizer on weights is set as 0.002.
CLTFP are trained based on Adamax optimizer (Kingma and Ba, 2014), we randomly select 10% training data as validation dataset to control earlystopping. The architecture of CLTFP are built upon Keras framework (Chollet, 2015). The structures and parameters for other methods are set according to the reports on corresponding papers.

Comparison results
In this paper, the mean absolute percentage error (M AP E) is used to compare the performance of traffic forecasting. The M AP E will be lower if the traffic volumes are higher. In observance of this, this paper also applies the mean absolute error (M AE) as a complementary measure for M AP E, M AP E = 1 n p n t=1,s=1 Corr(z :t , N :t ), where z :t = predicted traffic flow vector at time point t; N :t = actual traffic flow vector; n t = number of prediction steps. found that CLTFP achieves better performance than other methods in terms of prediction accuracy and spatial distribution. The reason is that CLTFP makes full use of spatial distribution, short-term temporal variability and long-term periodicities.

Analysis of Features
One constant criticism of using neural networks on transportation area has been that they are black box models, with little understanding of how the networks work and what knowledge the networks find from data. Recently, it is found that Granger Causality, which characterizes the causality based on incremental predictability, can be adopted to understand black-box typed prediction approaches . In this subsection, we focus on analyzing proposed CLTFP from the view of Granger Causality. The experimental analysis is conducted as following: we leverage available spatial features (S), short-term temporal features (T) and periodic features (P) generated from well-trained model CLTFP to train several Lasso models, and then conduct a comparison between forecasting results of those Lasso models with different combinations of features. All Lasso models are fit with Least Angle Regression, the penalty terms of l 1 priors are all set as 0.002.
The experimental results are given in Table. 2. This table is quite revealing in several ways. 1.
We can generally conclude that more types of feature help to build better prediction results. As the

Conclusions and future work
A novel deep learning based short-term traffic flow forecasting method CLTFP combined with CNN and LSTM is proposed in this paper, the forecasting results of CLTFP are encouraging, it indicates the potential of CNN and LSTM on transportation applications. Moreover, incremental predictability is applied to analyze the black-box typed forecasting method, the analysis shows that neural network based forecasting method can provide many meaningful knowledges of traffic flow.
This proposed CLTFP admits many improvements and extensions: 1. The features captured by LSTM achieves only modest forecasting accuracy, some more complex structures, for example, convolutional LSTM structure (Xingjian et al., 2015) can be an alternative.
2. Traffic flow are affected by many other factors such as weather, social event and state of the roads. How to exploit those information as auxiliary information is a future direction.
3. The applications of our model on general transportation network and similar spatial-temporal data on other domain are straightforward extensions.