{'fraction_non_alphanumeric': 0.033324853726787076, 'fraction_numerical': 0.0077406694043682086, 'mean_word_length': 4.737698081734779, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Geospatial Data to Images: A Deep-Learning Framework for Traffic Forecasting

Traffic forecasting has been an active research field in recent decades, and with the development of deeplearning technologies, researchers are trying to utilize deep learning to achieve tremendous improvements in traffic forecasting, as it has been seen in other research areas, such as speech recognition and image classification. In this study, we summarize recent works in which deep-learning methods were applied for geospatial data-based traffic forecasting problems. Based on the insights from previous works, we further propose a deep-learning framework, which transforms geospatial data to images, and then utilizes the state-of-the-art deep-learning methodologies such as Convolutional Neural Network (CNN) and residual networks. To demonstrate the simplicity and effectiveness of our framework, we present a formulation of the New York taxi pick-up/drop-off forecasting problem, and show that our framework significantly outperforms traditional methods, including Historical Average (HA) and AutoRegressive Integrated Moving Average (ARIMA).


Introduction
In recent years, smart city has been a new urban development vision, in which information and communication technologies are integrated with all kinds of components in a city, including the transportation systems. The success of the intelligent transportation system and smart city heavily relies on accurate traffic information as well as short-term traffic forecasting. By forecasting the traffic volume, travel speed, and occupancy of the next five to thirty minutes, traffic can be scheduled and planned in advance, which can alleviate traffic congestion and prevent traffic The prerequisite of traffic forecasting is the acquisition of accurate traffic information. Because of the development of sensing technologies and the popularity of smart devices, e.g., smart phones and tablets, we can collect and store large amounts of geospatial data, which include massive Global Positioning System (GPS) traces, smart card data, cell phone records, and users' locations and trajectories on social media, such as Facebook and Foursquare check-ins. On the premise of privacy protection, more datasets are being shared for research purposes, e.g., Geolife GPS trajectory dataset [1] , Caltrans performance measurement system database [2] , and New York trip record data [3] .
The large volume of these open datasets necessitates new processing methodologies. In recent years, deep learning has been an outstanding research field, because of the advent of new deep-learning models, increased chip processing abilities (e.g., graphics processing units), and widely available training datasets. Deep-learning methods have been used to solve various problems, such as object recognition [4] , handwriting recognition [5] , speech recognition [6] , image classification [7] , and machine translation [8] , and have resulted in a significant improvement over traditional methods.
Based on insights from the previous works, we further propose a deep-learning framework that transforms geospatial data to images, and then utilizes state-of-the-art deep-learning methodologies such as Convolutional Neural Network (CNN) [36] and residual network [37] . In our framework, geospatial data and external factors, such as weather condition, wind speed, and holiday information, are taken as inputs. Then we conduct data transformation and image preprocessing for the geospatial data and feature extraction for the external factors. The processed data are fed into the deep-learning models, which we then use to generate desired predictions.
To demonstrate the simplicity and effectiveness of our framework, we present a formulation of the New York taxi pick-up/drop-off forecasting problem and compare our framework with traditional models such as Historical Average (HA) and AutoRegressive Integrated Moving Average (ARIMA). We also investigate the influences of different parameters, e.g., the length of lookback window and the number of Residual Network Units (RNUs). The experimental results show that compared to the traditional methods, our framework can achieve a remarkable improvement in traffic forecasting.
In summary, our contributions in this study are as follows: We review works in which deep-learning models have been used to solve traffic forecasting problems.
We propose a deep-learning framework which transforms geospatial data to images before utilizing the state-of-the-art deep-learning methodologies and can be applied in different kinds of traffic forecasting problems.
We take the New York taxi pick-up/drop-off forecasting problem as a case study and demonstrate the simplicity and effectiveness of our framework over traditional methods.
The rest of this paper is organized as follows: Section 2 presents a short introduction to deep learning basics and a summary of previous works in which deep learning methodologies were applied to traffic forecasting problems. Section 3 describes the key components of our framework, including the processes of transforming geospatial data to images and the utilization of state-of-the-art deep-learning methodologies. Section 4 presents the New York taxi pick-up/drop-off forecasting problem as a case study and shows how we applied our framework to this problem. We present our conclusions in Section 5.

Deep-learning basics
Deep learning is a class of machine learning techniques, where many layers of information processing stages in hierarchical supervised architectures are exploited for unsupervised feature learning and pattern analysis/classification. The essence of deep learning is to compute hierarchical features or representations of the observational data, where the higher-level features or factors are defined from the lower-level ones [38] .
In this section, we briefly introduce Artificial Neural Networks (ANNs), CNNs, and Recurrent Neural Networks (RNNs), which are basic and widely used deep-learning concepts. A systematic introduction of deep learning is given in Refs. [38][39][40].

Artificial neural network
Artificial neural networks are learning models inspired by biological neural networks that approximate functions that depend on a large number of inputs (features or data representation) [41] . A computing unit or an ANN neuron usually consists of two components: the aggregation function, which calculates the sum of the inputs, and the activation function, which generates the outputs. Popular choices of the activation function include the logistic sigmoid, hyperbolic tangent (tanh), and Rectified Linear Units (ReLU). A specific example of ANNs is given in Fig. 1. If the neural network has the same number of nodes in the input and output layers, it is also called an autoencoder. The ANN shown in Fig. 1 is also an autoencoder.
A Deep Neural Network (DNN) is an ANN with multiple hidden layers between the input and output layers [42,43] . A DNN can be trained with the standard back-propagation algorithm [43] , which is a method of calculating the gradient of the loss function (to produce the cost associated with a given state) and set the weights iteratively. While DNNs can produce superior results, they are prone to overfitting because of the multiple hidden layers, and regularization methods are applied to combat overfitting [44] . Added layers also result in more training parameters, and various methods to speed up computation have been applied, such as batching (computing the gradient on several training examples at once rather than individual examples) [45] . In addition, the increased chip processing abilities (e.g., GPUs) alleviate the computation problem, as the matrix and vector computations required are well-suited for GPUs [43] .

Convolutional neural network
Convolutional neural networks [36] are designed for processing two-dimensional images (and other twodimensional representations), and the convolutional layer functions as a hidden layer, in which each group of neurons (also called filters) performs a convolution operation in the image and each neuron in a filter is connected to a different region of the image but the neurons share the same weights.
Because the input image may have a large resolution, max-pooling [46] is often used in CNNs to reduce the original size. Max-pooling applies a max filter to nonoverlapping regions of the initial image. The process of applying convolution and max-pooling operations is illustrated in Fig. 2.

Recurrent neural network
RNNs [48] are neural networks with loops. They can be trained by the reverse mode of automatic differentiation [49] , or the back-propagation algorithm by "unfolding" the network through time and constraining some of the connections to always hold the same weights [50] .
Long Short-Term Memory (LSTM) networks [51] are RNNs that solve the vanishing gradient problem [52] , when the gradients of some of the weights start to shrink or enlarge if the neural network is unfolded too many times. In an LSTM, the hidden layer is replaced by recurrent gates called forget gates (Fig. 3). Compared to RNNs, LSTM has been proved to generate superior results in many problems.

Traditional methods
Short-term traffic forecasting has been an active research area for over sixty years. Before the popularity of deep learning, traffic forecasting-related problems have been addressed with many models, which include the following: Time-series models, such as ARIMA [54] , in which the future value of time series is a linear combination of previous values and residuals, and differentiation is used to obtain stationarity from the nonstationary time series [55] ; Vector AutoRegressive (VAR), which captures the linear interdependencies among interrelated time series [54] ; and seasonal ARIMA, which applies the additional seasonal difference to seasonal time series before using ARIMA [56] . Fig. 2 Convolutional and max-pooling operations of a convolutional neural network using a 3 3 filter (adapted from a deeplearning tutorial [47] ). State-space models, such as the spatiotemporal random effect model [57] , which use a set of inputs, outputs, and state variables to describe a system by a set of first-order differential equations.
Non-parametric models, such as Bayesian network [58] and spatialtemporal weighted k-nearest neighbor [59] , which are data-driven and approximate the relationship between the dependent and independent variables using a training dataset.
The traditional methods are adequately studied and described in Ref. [60].

Summary of previous works
With the developments of deep learning and more open geospatial datasets in recent years, deep-learning models have been trained with big data and then applied to different traffic forecasting problems. In Table 1, we list some recent works in which deep-learning Table 1 A summary of previous works.
In Table 1, we categorize the previous works by the problems they address, and then list the used datasets, deep-learning methods, and baseline methods. For the deep-learning methods, we list the specific family of models being used, such as LSTM, where possible; otherwise, we list a more general concept, Table 2 Abbreviations used in Table 1 and their corresponding terminologies (in alphabetical order).

Geospatial Data to Images: Our Framework
Based on insights from previous works, we propose a deep-learning framework that transforms geospatial data to images before utilizing state-of-the-art deeplearning methodologies. Our framework can be summarized as shown in Fig. 4. Our framework has two inputs: geospatial data and external factors. For the geospatial data, we perform data transformation and image preprocessing to transform geospatial data to images. For the external factors, we extract their features. Then both the images and features are fed into the deep-learning models to yield the predicted results.

Data transformation
In this section, we introduce two methods of transforming geospatial data to images: individualbased data transformation and crowd-based data transformation. The choice between these two data transformations depends on the specific application. For both methods of data transformation, the geospatial data need to be transformed to gray images. To effect this, we have to grid the whole map based on the location information of the geospatial data, and then each grid cell is seen as a pixel. Intuitively, a smaller gap between grids would generate an image with a higher resolution. We also have to partition the time domain into small time intervals. Because we are dealing with forecasting problems, the common scenario is to predict the situation of the next time interval based on the data collected in the previous time intervals. Then using the data within each cell, we can transform a cell into a single value by adding the data or taking the average, or other application-centric methods.
To explain the transformation process, we use an example of check-ins collected from two individuals, within a location range of OE0; 3/ OE0; 3/ and time range of OE0; 3/, as shown in Figs. 5 and 6.

Individual-based data transformation
For individual-based data transformation, the geospatial data are grouped by different individuals and transformed to an image per individual. As shown in Fig. 5, the value in each cell represents a 0/1 binary variable which indicates whether the individual has visited the cell or not. With other geospatial data, it may also be a continuous variable, such as the moving speed of the individual in the cell.
Individual-based data transformation is suitable for individual-based applications, such as human trace forecasting and taxi destination forecasting.

Crowd-based data transformation
For crowd-based data transformation, the geospatial data are grouped by different time intervals and transformed to an image per time interval. As shown in Fig. 6, the value in each cell represents the number of check-ins within a cell in a specific time interval. With other geospatial data, it may also represent other crowdbased characteristics, such as the size of the flow, the number of individuals, and the supply and demand of transportation services.
Crowd-based data transformation is suitable for crowd-based applications, such as traffic flow forecasting and crowd flow forecasting.

Image preprocessing
Before feeding the images into the deep-learning models, some preprocessing operations may be applied, such as image fusion and image selection, to choose the appropriate images to use.

Image fusion
Image fusion has been used in computer vision area to combine two or more images from different sources into a single image [61] . One advantage of image fusion is that the combined image could provide more information than any of the input images. In addition, the errors in a single image can be avoided by comparing with other images, if the images overlap in spatial and temporal domains.

Image selection
While forecasting for crowd, a practical method is to choose a sliding lookback window and use the data within this lookback window to predict the future case. This corresponds to an image selection process where only the most recent images are used to predict the near future. Based on the temporal properties, different images may be selected, such as images in the past few hours and images in the same time interval of the previous weekday.

Feature extraction
Traffic conditions are affected by external factors other than the geospatial data, such as weather data and holiday information. It has been known that traffic situations are highly affected by the weather, and the traffic patterns in workdays and holidays are different; for example, there is an obvious daily commute pattern on workdays. We need to incorporate this effect into our predictions.

Deep-learning models
The choice of deep-learning models involves domain knowledge as well as trial and error. Although different deep-learning models or their combinations can be tried and applied for forecasting problems, CNNs and RNNs are widely used to capture the spatial and temporal dependencies among the geospatial data. The performances of the deep-learning models may depend on the specific problems. In the next section, we give a specific example of the deep-learning model used in the New York taxi pick-up/drop-off forecasting problem.

New
York Taxi Pick-up/Drop-off Forecasting Problem In this section, we define the taxi pick-up/drop-off forecasting problem based on a New York trip record data [3] , which we use to formulate the problem and validate our framework. The taxi pick-up/drop-off in one area may be affected by the neighborhood; for example, when a taxi drops off a passenger, it may search for passengers in nearby areas. The pickup/drop-off events of taxis are also affected by the most recent time intervals and may exhibit a periodical pattern; the pick-up/drop-off during morning/evening rush hours may be similar on consecutive workdays or the same day of the previous week.
For this problem, we supported our deep-learning model with CNNs and residual networks and compared it with time-series models, such as HA model and ARIMA, which capture temporal dependencies. Based on our experiments, our deep-learning model achieved a remarkable improvement over HA model and ARIMA.

Dataset description
In our experiments, we used the NYC taxi trip record data provided by the NYC Taxi and Limousine Commission, covering both yellow and green taxis. Unlike yellow taxis, green taxis can only pick up passengers in upper Manhattan and the outer boroughs. Each taxi trip record includes fields capturing pick-up and drop-off dates/times, pick-up and drop-off locations (longitudes and latitudes), trip distances, itemized fares, rate types, payment types, and driver-reported passenger counts. We mainly used pick-up/drop-off timestamps and locations of taxi trips from July 2015 to June 2016, which contain 138 413 407 yellow taxi trips and 18 355 786 green taxi trips.

Data transformation and problem description
We used crowd-based data transformation in our taxi pick-up/drop-off forecasting problem. To realize this, we partitioned the New York map into an M N grid map based on the location information of the geospatial data, such as the longitude and latitude, where each square formed by grids is referred to as a cell. For the high-level feature, we aimed to predict the taxi pickup/drop-off number within cell .i; j / in time interval k, based on the previous data.
Formally, we denote a taxi trip record r as where t O r and t D r represent the pick-up and drop-off timestamps, respectively; x O r and y O r represent the pick-up longitude and latitude, respectively; x D r and y D r represent the dropoff longitude and latitude, respectively. For each cell, the longitude and latitude ranges of cell .i; j / were represented as OEx i ; x i C1 / OEy j ; y j C1 /. Time interval k was defined as OEt k ; t kC1 /. We define the taxi pickup/drop-off numbers within cell .i; j / in time interval k as follows: v r < x i C1 ; y j y D r < y j C1 gj; where j j represents the cardinality of a set, r represents a trip record, and R represents the set of all the trips (including yellow and green taxi trips).
We denote the taxi pick-up/drop-off numbers within time interval k as a tensor V k 2 R 2 M N , where .V k / 0;i;j D v pick-up i;j;k , .V k / 1;i;j D v pick-up i;j;k . The taxi pickup/drop-off forecasting problem can be stated as given fV k jk D 0; 1; : : : ; n 1g, predict V n .
We used Root Mean Square Error (RMSE) as our evaluation metric, which can be formulated as where O v and v are the predicted and true values, respectively, and S is the total number of the values to predict.
In our experiments, we partitioned the New York map into a 25 25 grid map and evenly split a day into 48 time intervals. We used the taxi trip record from July 2015 to May 2016 as the training set and the taxi trip record in June 2016 as the testing set. We evaluated the RMSE for pick-up and drop-off predictions separately.
To prepare the inputs for the deep-learning model, we conducted two more preprocessing operations: We used the log function to transform the scale of the original pick-up/drop-off numbers, as the supply and demand for taxis was unbalanced in different areas, and thus, the distribution of these numbers was highly skewed. Most cells had a small number of pick-ups or drop-offs, and only few cells had an extremely large number; Then we used a min-max normalization process to scale the numbers into the range OE 1; 1, as required by the tanh activation function of the deep-learning model.
For evaluation, the reversed processes were performed before the RMSEs were calculated.

Image preprocessing
Since we lacked other sources of data which also reflects the taxi pick-up and drop-off situations, image fusion technologies were not used. In the image selection process, to predict the pick-up and drop-off numbers V k 2 R 2 25 25 in interval k, we chose the following pick-up and drop-off numbers as inputs: V k `; : : : ; V k 1 , where`represents the length of the lookback window; V k 48 , which represents the pick-up/drop-off numbers in the same time interval a day earlier; V k 7 48 , which represents the pick-up/drop-off numbers in the same interval a week earlier; External factors, including weather, wind speed, and holiday data.
We compared the effect of different lengths of the lookback window (i.e.,`D 2; 4; 6).
While our target was to predict the overall taxi pickup/drop-off situation, there were two types of taxis: yellow taxis and green taxis. Although we could easily add the pick-up/drop-off numbers from these two types of taxis, it would be interesting to investigate if a better prediction result can be achieved by separately using the pick-up/drop-off images from the yellow taxis and green taxis separately. This would be considered in the future.

Feature extraction
External factors, such as weather and wind speed, may affect the demand for taxis. On a rainy or windy day, the demand for taxis may increase as those who usually use bicycles may find it inconvenient to ride in a bad weather.
To prepare the external factors as inputs for the deeplearning model, we used one-hot coding to transform the holiday data, and weather conditions into binary vectors, where 1 indicates the day is a holiday, and 0 indicates otherwise. In addition, we had a 25-bit vector for the 25 types of weather conditions, such as rain, snow, and fog. The temperature and wind speed were scaled to the range OE 1; 1 with the min-max normalization. Because we lacked data on the weather condition, temperature, or wind speed of the future time interval k we aimed to predict, we only used the information of the most recent time interval k 1. All these vectors and scaled values were then concatenated to a long vector and used as the input for the deeplearning model.

Deep-learning model
In our deep-learning model, we combine CNNs and residual networks. Residual networks have been used to build extremely deep models and achieve a tremendous success on tasks such as image recognition [37] .
In our implementation of the deep-learning model, we used Theano [62] , which is a Python framework for fast computation of mathematical expressions, and Keras [63] , which is a high-level neural network application programming interface written in Python. For convenience, we used the Keras layers to present our deep-learning model as shown in Fig. 7. Each layer used in the model is explained as follows: InputLayer: the layer that specifies the input; Conv2D: the two-dimensional convolutional layer, in which each filter has a size of 3 3; Activation: the layer that applies an activation function to an output, e.g., the ReLU function and tanh function; Merge: the layer that merges a list of inputs; Dense: the regular densely-connected neural network layer; Reshape: the layer that reshapes an output to a certain shape.
While the input images were independently processed at first, a merge layer was used to fuse the influence of different images as well as the external factors. In this way, the influence of different images, which represents the pick-up/drop-off situation in the past few hours, a day earlier, and a week earlier, could be learned separately and their degrees of influence on the time interval k could be quantified and adjusted by the merge layer.
Since we were predicting the taxi pick-up/drop-off numbers within the same cells with the inputs, we did not use subsampling or max-pooling in the CNN for this specific problem, but only convolutions. For the residual network, we used an Activation layer and a Conv2D layer as an RNU, as shown in Fig. 7. We studied the effects of different numbers of RNUs used in the model.
The loss function to be minimized in the training process is the mean squared error between the predicted pick-up/drop-off numbers and the true pick-up/drop-off where Â denotes all learnable parameters in our deeplearning model, O V k denotes the predicted pick-up/dropoff numbers and V k denotes the true pick-up/drop-off numbers, both in time interval k.
Algorithm 1 outlines our training process. To accelerate the training process, we use the mini-batch technology [64] (we use 25 as the batch size) and Adam [65] as the optimizer.

Baseline methods
We compared our deep-learning model with two baseline methods: Historical Average. In our implementation of the HA method, we used the weekly average values of pickups and drop-offs in each time interval.
ARIMA. We used a Python module statsmodels for our implementation of ARIMA model and the parameters were optimized by grid searching.

Experiments and results
Our results of predicting the taxi pick-ups/drop-offs put an training instance .fI k ; E k 1 g; V k / into D 5: end for 6: initialize the parameter Â 7: repeat 8: randomly select a batch of training instances D b from D 9: find Â by minimizing the objective (1) with D b 10: until stopping criteria is met 11: return the trained model M using different methods and parameters are summarized in Table 3.
Our key findings are as follows: For the baseline models, HA captured the weekly pattern, while ARIMA used the most recent information. Based on our results, the weekly pattern outstripped the most recent information in the taxi pickup/drop-off forecasting problem.
Our deep-learning framework yielded a remarkable improvement over both baseline methods.
With different lengths of lookback window`, we may achieve different performances for pick-up or drop-off forecasting. As seen from our results, using a smaller`of 2, the best RMSE for pick-up forecasting was achieved, and using a bigger`of 6, the best RMSE for drop-off forecasting was achieved.
Using more RNUs does not always result in better performance. When the length of lookback window`is fixed as 4, the best performance was achieved by using only one RNU.

Conclusion
In this study, we focus on the application of deeplearning technologies in a series of traffic forecasting problems based on geospatial data. In addition, we propose a preliminary framework for traffic forecasting, which transforms the geospatial data to images and leverages the state-of-the-art deep-learning technologies. Our experiments on New York taxi pickup/drop-off forecasting problem demonstrate that our framework greatly outperforms traditional methods.