{'fraction_non_alphanumeric': 0.03288276579152587, 'fraction_numerical': 0.005003899142188718, 'mean_word_length': 5.108971814211989, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Large-Scale Traffic Congestion Prediction based on Multimodal Fusion and Representation Mapping

With the progress of the urbanisation process, the urban transportation system is extremely critical to the development of cities and the quality of life of the citizens. Among them, it is one of the most important tasks to judge traffic congestion by analysing the congestion factors. Recently, various traditional and machine-learning-based models have been introduced for predicting traffic congestion. However, these models are either poorly aggregated for massive congestion factors or fail to make accurate predictions for every precise location in large-scale space. To alleviate these problems, a novel end-to-end framework based on convolutional neural networks is proposed in this paper. With learning representations, the framework proposes a novel multimodal fusion module and a novel representation mapping module to achieve traffic congestion predictions on arbitrary query locations on a large-scale map, combined with various global reference information. The proposed framework achieves significant results and efficient inference on real-world large-scale datasets.


I. INTRODUCTION
With the rapid development of the automobile industry in recent decades, the production and ownership of motor vehicles have increased significantly. At the same time, the increasing traffic demand is imposed on the urban transportation system due to the accelerated urbanisation process. Especially during peak hours, the road capacity of some road sections cannot meet the huge traffic demand, resulting in congestion problems. Severe traffic congestion may lead to extra carbon emissions and reduce the transportation network efficiency, even bring huge economic losses [12]. Therefore, it has always been regarded as one of the most important traffic problems by analysing the factors of congestion and establishing traffic congestion prediction. Traffic congestion prediction plays an important role in a wide range of applications where: 1) Advanced Traffic Management Systems (ATMSs) and Advanced Traveller Information Systems provide real-time guidance information to travellers; 2) individuals or authorities optimise resources allocation regarding the required traffic time to ensure the journey smooth for travellers; 3) governments and urban planners design road network expansion plans [40].
In the early decades, due to the lack of data availability, traffic prediction was biased towards predicting the traffic parameters of individual roads and local networks. Researchers at that time preferred to apply classical statistical models to predict traffic volumes. However, with the development of traffic, roads tend to be networked, and traditional statistical models can not well address the complexity of traffic data [6]. However, the rapid advancement of artificial intelligence and machine learning has empowered a new wave of evolutionary solutions to many previously intractable engineering problems. Machine learning models can automatically adjust predictive model parameters to compensate for the shortcomings of traditional models, and thus are widely used in the field of traffic flow prediction. At present, the mainstream prediction models based on machine learning are Recurrent Neural Network (RNN), Convolutional Neural Network(CNN), and Factorisation Machine supported Neural Network (FNN) [29]. As one of the classical neural network models, FNN's fully connected structure enables it to process all functional combinations of the previous layer, but this connection structure also makes the training process very time-consuming [6]. The RNN-based LSTM model has achieved very good results in the field of time series forecasting of traffic flow forecasting models. However, when the input data is a high-resolution twodimensional data sequence, the processing speed of LSTM is slow [34]. Unlike RNN and FNN, CNN does not have any difficulty in handling high-resolution data, and its local connection structure also makes the training process shorter than the other two methods.
While the availability of big data permits researchers to explore this field further, different factors that affect traffic congestion, e.g., social media, economic state, and urban layout elements, are not fully considered. As a typical complex system of the urban road network, reasonably considering the interaction between different factors can effectively improve prediction accuracy. Also, incorporating diverse factors from the above-mentioned aspects for traffic congestion prediction is still unexplored. Therefore, aiming at traffic congestion prediction, designing a complex framework based on multimodal fusion and representation mapping becomes urgent and necessary with time as larger datasets can be accessed.
In order to better fuse the rich information on a large-scale map and infer traffic situations, a novel and efficient framework is proposed in this paper so that the global multimodal information referenced by traffic prediction is aggregated into a geo-preserving representation in a high-dimensional superspace, which can be utilised with mapped locations to predict traffic situations at different locations and time points (Fig. 1).
The main contributions of this paper are summarised as follows: • A novel and lightweight end-to-end traffic congestion prediction framework based on representation fusion, which can use diverse multimodal global information from a large-scale map as the reference to predict traffic congestion at precise geographic locations and different time slots efficiently. • A global meta-representation learning method that generalises the multimodal global reference information on the corresponding spatial geographic structure of a largescale map, learned by a Multimodal Fusion and Generalisation Module. • A geographical query mechanism based on a Query Location Mapping Module, which enables efficient traffic congestion inference through learning and mapping location representations with the module. • Intensive experiments and comprehensive analysis on real-world large-scale geolocated datasets, which demonstrate the effectiveness of the proposed method and stimulate further applications for road traffic planning.

A. Traffic Congestion Prediction
The field of traffic forecasting has existed for nearly fifty years, and most of the initial research originated from civil engineering and traffic engineering research. However, with the rise of supercomputers and artificial intelligence technology, traffic flow prediction has gradually become interdisciplinary research. Mainstream traffic flow forecasting models are mainly divided into three categories. The first category belongs to classical statistical models, of which the Auto Regressive Integrated Moving Average (ARIMA) family of models is the most widely used [29]. ARIMA was first applied in transportation prediction in 1979 [1]. It was then applied to traffic flow prediction on highways by Levin and Tsao and reached the conclusion with the highest statistical significance of ARIMA(0,1,1) [16]. With the development of urban traffic, traffic data tends to be nonlinear and big data, and traditional statistical models are no longer applicable [15], [17]. Due to the shortcomings of classical statistical models, the researchers start to pay attention to non-parametric models, k-nearest neighbour (KNN), support vector machine (SVM), Bayesian network (BN), and artificial neural network (ANN) [3], [25], [28], [36]. Machine learning is gradually applied to the field of traffic flow prediction. Chan et al. proposed a new neural network training method using a hybrid exponential smoothing method and the Levenberg-Marquardt (LM) algorithm, aiming to improve the generalisation ability of traffic flow prediction methods previously used to train neural networks. The improved neural network training method can denoise the traffic flow data [4]. Cai et al. proposed an improved KNN model to improve prediction accuracy based on spatiotemporal correlations, enabling multi-step prediction. Unlike the previous KNN model that uses time series to describe the traffic state of the road network, the improved KNN model uses the spatiotemporal matrix to describe the traffic state.
While each model has its advantages and can achieve good forecasting under different traffic conditions, It is difficult for a single model to cope with complex situations in the road network system. Zhou et al. proposed a deep learningbased multimodal learning framework to predict future traffic flow conditions based on current input data by dynamically selecting an optimal model or a subset of optimal models from a set of candidate models [41]. Based on the ensemble of Gradient Boosted Regression Tree (GBRT) and Least Absolute Shrinkage and Selection Operator (Lasso), Chen et al. proposed an ensemble learning algorithm for short-term traffic flow prediction, which takes into account accidents, accidents, adverse weather, work zones, and holidays [5]. The previous multimodal traffic prediction models mostly considered the fusion of different models but seldom considered the impact of different data types on the results.

B. Multimodal Representation Learning
Since information of different modalities such as texts, figures, audios, videos are increasingly appearing in diverse applications, such as video classification [33], object recognition tasks [26], egocentric activity recognition [10], etc., learning fusion representations of modalities has been going on for decades. There are three main types of methods to represent multimodal fusion. First, the joint representation, which integrates unimodal representations through mapping them together into a unified space. Based on joint representation, many classification or clustering tasks have been applied, such as event detection [32] [14], sentiment analysis [22] [37], and visual question answering [13]. The second category is coordinated representation, which take advantages of cross-modal similarity models to represent unimodal in a coordinated space separately. The last one is encoder-decoder model, which aims to acquire intermediate layers that project one modality to another. This framework has been widely exploited for multimodal translation tasks, such as image caption [35] [31], video description [8] [30], and image synthesis [23].
In recent years, deep learning based multimodal representation learning has received a lot of attention and research. Srivastava and Salakhutdinov [27] proposed a Deep Boltzmann Machine model as a generative model, which can extract a unified representation that fuses modalities together by learning the joint space of image and text inputs. These raw inputs are transformed into corresponding high-level representations, then the model learns joint distribution of each representation. They found that if is useful for classification and information retrieval tasks from both unimodal and multimodal queries.
For multimodal data fusion, Ngiam et al. [21] proposed Stacked AutoEncoder (SAE). The model not only gets a solu-tion to cross-modality by obtaining better unimodal representation from knowledge of other modalitiesbut also learns the complicated correlation between modalities, with intermediate levels being learned as shared modality represent learning. To exploit the matching relationships between image and text modalities, Ma et al. [19] proposed a multimodal convolutional neural network (m-CNN). By using an image CNN to grasp image content with a corresponding matching CNN, the joint representation of images and sentences is captured. To be specific, the matching CNN combines several semantic levels of words, from word level, to phase level, and sentence level. For image captions generation, Mao et al. [20] proposed a multimodal recurrent neural architecture (m-RNN). The model contains two parts, one is deep recurrent neural networks for learning sentences and another is deep convolutional neural networks for understanding images. These two networks are in constant interaction with each other throughout the multimodal network.
In this study, spatial distributions from three different sources, including text and other inputs are learned and convolution operation is implemented on three modalities. The joint representation of sentences and numerical values are learned from 2D map space. Finally, these joint features are used to predict the classification of traffic congestion based on location query. Thus, the distribution of location query and multimodal representations are converged into a unified mapping space.

A. Overview
Current traffic prediction models are mostly based on deep learning. However, the processing of FNN is too timeconsuming, while it is too difficult for RNNs to deal with highresolution data. To step aside these limitations, a novel and lightweight end-to-end traffic situations prediction framework based on representation fusion is proposed, using CNNs to predict traffic congestion sequence at any location on a largescale map efficiently, which employs multimodal social data with the location of the large-scale map as global reference information.
Specifically, the global reference information is divided into three categories (social media texts, real estates, and points of interest), which are scattered across the whole map with different distributions respectively (Fig. 2). Also, being formatted learnedly and adaptively, the multimodal global reference information is learned and fused into a global-aware geopreserving representation, also called meta-representation, for aggregating all reference information of the entire large-scale map. In addition, a geo-preserving representation is learned to match a specific location used to capture traffic congestion situations. Finally, the global-aware representation and the location-aware representation are aggregated and learned to obtain location-specific traffic congestion results.

B. Problem Statement
A traffic congestion prediction task can be formulated to utilise the proposed method to obtain valuable results. Firstly, the entire large-scale map is gridded to characterise all the data. As a result, the original map is formatted as a 2dimensional grid array whose size is H × W . For each grid on the map, three kinds of global reference information in the region are quantised as multi-dimensional features that are located in the correlated location of the whole gridded map, which is also regarded as three multi-channel matrices.
Specifically, for the region represented by each grid, a vector with a length of D mt is obtained to represent the social media information from the dataset after pre-processing. For grids without any social media information, zero-vectors are used. Similarly, for a grid, pre-processed real estates information and points of interest information can be represented by a D re -length vector and a D pi -length vector respectively. As a result, for the entire gridded map, social media texts, real estates, and points of interest information can be formulated as three multi-channel matrices: F mt , F re , and F pi , where F mt ∈ R H×W ×Dmt , F re ∈ R H×W ×Dre , and F pi ∈ R H×W ×Dpi , which are employed as a part of input of the proposed framework.
For the gridded map of size H × W , different regions are divided into training, validation, and test regions, respectively, while the data in these regions are aggregated into training, validation, and test sets. With the help of reference information as well as traffic data in the training area, the task is to predict whether roads are congested in any unvisited grid area (test area). For the whole gridded map, the index of each grid x h,w is the location of the hth row and the wth column, where h = 1, 2, ..., H, and w = 1, 2, ..., W . For each x h,w , the proposed model is able to obtain the results,ŷ h,w , which is a vector with multi-length for describing whether the corresponding location is congested at multiple time points in a day. To this end, the proposed method is expected to minimise the difference between the predictedŷ h,w and the ground truth of the location, which is denoted as y h,w .

C. Framework
For the traffic congestion prediction task, the proposed novel and lightweight framework learns and generalises the global reference information well, so that the matching and fusion of specific query points are able to handle significant predictions. To be more specific, as shown in Fig. 3, featured global reference information (F mt , F re , and F pi ) of the whole largescale map (Part. A in Fig. 3) is firstly fed into Multimodal Fusion and Generalisation Module (MFGM) (Part. B in Fig. 3) for obtaining the Global-aware Meta-representation (Part. C in Fig. 3) of the map. At the same time, a query location (x h,w ) is transferred into a Location-aware Mapped-representation (Part. E in Fig. 3) via Query Location Mapping Module (QLMM) (Part. D in Fig. 3). Finally, the Representation Fusion and Prediction Module (RFPM) (Part. E in Fig. 3) fuses the Globalaware Meta-representation and the Location-aware Mappedrepresentation and makes prediction via a neural network.
For each step in training, all features of the global reference information are always fed into the MFGM. As a result, when the entire model is trained to convergence, MFGM is robust enough to extract and generalise multimodal global reference information comprehensively and represent it as the Global-aware Meta-representation. Therefore, while inference, as an adapted representation of the whole map, a stable and informative Global-aware Meta-representation that is saved after training is directly fed into the RFPM without using the heavy MFGM architecture and the complex global reference information. In this way, the used framework maintains lightweight parameters and incredible efficiency.
The details of each module in the proposed framework will be introduced below: Then the saved robust Global-aware Meta-representation (shown as a normalised heatmap) is fused with generated Location-aware Mapped-representation for using as an input to make prediction via a well-trained RFPM.

1) Multimodal Fusion and Generalisation Module (MFGM):
For different types of global reference information which is formatted into the matrix with different channels, different convolutional neural networks are designed for obtaining three unimodal representations that have the same size, H ur × W ur × D ur . As shown in Part. B of Fig. 3, there are three neural networks denoted as F uni = f uni m (·) m=mt,re,pi so that the obtained unimodal representations of three types of the global reference information is U = {U m } m=mt,re,pi , where any U m ∈ R Hur×Wur×Dur , and the forward processing can be formulated as: A channel-wise concatenation is achieved over the three unimodal representations causing a joint representation fed into a convolutional neural network, f multi (·), similar to an Auto-Encoder. f multi (·) mixes representations from different modalities and moulds them into a global reference-rich multichannel representation with the number of channels D meta that preserves the spatial structure of the original map on each channel. The resulted representation is named as the Globalaware Meta-representation (Part. C in Fig. 3) and denoted as R meta , where R meta ∈ R H×W ×Dmeta , which is obtained by: 2) Query Location Mapping Module (QLMM): For each query location in the entire gridded map, it is difficult to describe the spatial relationship between the location and the whole map well in the form of coordinate pairs (x h,w ). In contrast, generating a learnable geo-preserving representation is a reasonable approach for carrying the information of the location. More specifically, a location-rich multi-channel matrix that can adapt the location to the global information can better participate in the whole framework to help the neural network make predictions because the multi-channel matrix can spatially hold the interaction between the current query location and all other locations relative to the global information. The multi-channel matrix is named as Location-aware Mapped-representation (Part. E in Fig. 3) and denoted as R loc h,w for each x h,w . However, with such complex high-dimensional interactions in geo-spatial, the distribution presented by the Locationaware Mapped-representation is not resolvable, therefore, a simple distribution is assumed as prior information of the Location-aware Mapped-representation. More precisely, in the initial state, the influence of the query location on the entire map spreads from the location to the surroundings in a two-dimensional Gaussian distribution. In other words, the probability that each location is affected by the query location follows a two-dimensional Gaussian distribution. Therefore, for each query location x h,w , affected probability value of the grid at location (h , w ) is denoted as m (h,w)→(h ,w ) , and where (a, b) are two-dimensional random variables and (a, b) ∼ N (x h,w , Σ). As a result, for each query location x h,w , a matrix of the same size as the original gridded map can be obtained as: Then M h,w can be normalised for constructing a Gaussian location mask M h,w (shown in Fig. 4). Based on the Gaussian location mask as a prior, the Location-aware Mappedrepresentation can be learned as a posterior via a convolutional neural network (shown in Fig. 4), f map (·) and the forward processing is shown in Part. D of Fig. 3 and formulated as: 3) Global-Location Representations: As shown in Part. C and Part. E of Fig. 3, there are two key representations generated in the framework: Global-aware Meta-representation (R meta ) and Location-aware Mapped-representation (R loc h,w ). In the proposed framework, Global-aware Meta-representation and Location-aware Mapped-representation are set to have the same size as the original gridded map for each channel. Also, these two representations have high-dimensional channels. In this way, for a query location, each grid has two highdimensional vectors representing global reference data and spatial relations respectively, so that they can carry information for making predictions significantly.
Furthermore, the Global-aware Meta-representation is representing all of the global reference information, so that it can be utilised as a generalised feature map and has the potential to complete multiple tasks.

4) Representation Fusion and Prediction Module (RFPM):
A channel-wise fusion is implemented on Global-aware Meta-representation (R meta ) and Location-aware Mappedrepresentation (R loc h,w ), which causes feeding for a neural network with a fully connected output layer, f pred (·), in order to obtain the prediction resultsŷ h,w for the query location x h,w . The outputŷ h,w is a multi-dimensional vector whose single element represents the congestion at a time point of the monitored time series. The forward propagating is illustrated in Part. F of Fig. 3 and formulated as:

D. Optimisation and Efficient Inference
When the model is trained, for each query location x, a temporal sequence of traffic congestion valuesŷ h,w with T time points is obtained, whereŷ h,w = [ŷ 1 h,w ,ŷ 2 h,w , ...,ŷ T h,w ], and the Mean Squared Error (MSE) is used as the objective function to measure the loss betweenŷ h,w and the ground truth y h,w , where y h,w = [y 1 h,w , y 2 h,w , ..., y T h,w ]. As a result, the loss can be formulated as: Based on the above objective function, the neural networks in MFGM, QLMM, and RFPM are trained to converge step by step. Since the global reference information is diverse and complex, MFGM is bulky and heavy. However, after the model converges, the Global-aware Meta-representation is preserved and can represent the massive global reference information stably. Therefore, when training is completed, only the lightweight neural networks in QLMM and RFPM are utilised so that the framework can achieve fast and efficient inference. More specifically, as shown in Fig. 5, the Locationaware Mapped-representation generated by QLMM is directly fused with the saved Global-aware Meta-representation and then fed into RFPM to complete the prediction.

IV. EXPERIMENTS A. Implementation details
The experiments are mainly conducted on H4m dataset [38], [39], which consists of 28, 550 real estate, 497, 256 points of interest, 250, 000 traffic records and over 100 million pieces of social media text geolocated in the same city. As for the different kinds of data, real estate, social media text, and points of interest are set as the global reference information. Also, the reference information is pre-processed for featuring it into multiple metrics. For each kind of reference information: • Social media texts: All retrieved natural language texts sent out on social media are fed into a pre-trained BERT [7] to obtain a multi-dimensional vector of the grid with respect to social media texts. After the multidimensional vectors of all grids are obtained, a pre-trained Auto-Encoder [24] is used to reduce the dimensionality of the obtained vectors. Finally, the multi-channel matrix is obtained to be used as a part of the training data of the experiments. • Real estates: The average price of all real estates in each grid reflects the living quality of the area, so a normalised average real estates price is used as the real estates vector of the grid. And for grids without real estates information, zero is filled. Finally, for the entire gridded map, a singlechannel matrix is obtained as real estates features. • Points of interest: There are 23 categories of points of interest in the grid, whose diversity constructs the rich interest information of the current grid. Therefore, countbased one-hot encoding is used to vectorise the interest points of the grid. As for the whole gridded map, points of interest features are represented as a multi-channel matrix for experiments. At the same time, the traffic records are divided into training, validation, and testing set according to geographic locations, where the results on the testing set are reported. For more details, with in 250k grids on the whole map, there are around 24k grids total which are recorded with traffic data. For each grid, traffic data for 216 time points of one day is stored. All grids are divided into the training set, validation set, and test set at the ratio of 80%, 20%, and 20%, respectively. What is more, the traffic records are divided into two categories: congestion and no congestion, so that Accuracy, Recall, Precision, and F1-Score are used as metrics for evaluating the models.

B. Results
After the entire framework is trained and validated on the selected dataset, quantitative results are obtained from the evaluation on the testing set. Also, a KNN-based traffic prediction method [18] is referenced as the baseline model.  results for three blocks in the testing set at two time points are visualised in Fig. 6. As can be seen from Tab. I, all metrics of the main experiment are better than the baseline, which adequately demonstrates the effectiveness of the proposed framework. As shown in Fig. 6, the congestion of most roads is correctly predicted, regardless of whether it is a wide road or a narrow road. According to the comparison between the baseline results and the main experimental results, it is difficult to obtain meaningful prediction results by simply learning and searching based on the simple geographical relationship between different roads on the large-scale map, which proves the necessity and effectiveness of the proposed framework.

C. Ablation Study
In order to demonstrate the effectiveness of the proposed modules, some experiments are designed by leaving parts of the whole framework unemployed while remaining the global structures. To investigate the role of the Global-aware Meta-representation and the Location-aware Mapped-representation on the effect of the overall framework, these two modules are used separately and evaluated to obtain the results. More specifically, for deactivated Global-aware Meta-representation, a multi-channel zero-matrix of equal size is used instead and fused with the Location-aware Mapped-representation; for the Location-aware Mapped-representation, a weak mappedrepresentation is obtained by using a one-hot location mask instead of the Gaussian location mask in QLMM.
As can be seen from the evaluation results in Tab. II, both Global-aware Meta-representation and Location-aware Mapped-representation are indispensable in the proposed framework. It is worth noting that the posterior information learned from the Gaussian location mask designed based on the Gaussian distribution plays a very important role, and at the same time, the reference information included in the Globalaware Meta-representation is also decisive in prediction.
At the same time, the effect of multimodal input on the global information aggregation of the framework is also tested ablatedly. For the unimodal representations generated by the three kinds of global reference information (social media texts, real estates, and points of interest), their independent active states (denoted as M odel mt , M odel re , and M odel pi ) are used to evaluate the entire framework respectively. As shown in Tab. III, all metrics under the fusion of multimodal representations are optimal except for secondly ranked precision, which indicates that multimodal global reference information with MFGM works better than any unimodal information for enabling the proposed framework to make significant predictions.
V. CONCLUSION In this paper, an efficient convolutional neural networkbased end-to-end framework is proposed, combining multimodal fusion and representation mapping to address the traffic congestion prediction problem on large-scale maps.
To achieve the expected prediction, a module called Multimodal Fusion and Generalisation Module (MFGM) is designed to aggregate global reference information to obtain the Global-aware Meta-representation for the entire map. At the same time, for a query location, a Location-aware Mappedrepresentation is obtained to describe the current query location relative to the global response via the Query Location Mapping Module (QLMM).
Finally, the Global-aware Meta-representation and the Location-aware Mapped-representation are fed into the Representation Fusion and Prediction Module (RFPM) and get traffic congestion predictions for the query location. Different contrast and ablation experiments are implemented on the proposed framework to verify its advancements and superiority.