{'fraction_non_alphanumeric': 0.03124401913875598, 'fraction_numerical': 0.009282296650717703, 'mean_word_length': 4.770568746548868, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
RAP-Net: Region Attention Predictive Network for Precipitation Nowcasting

Natural disasters caused by heavy rainfall often cost huge loss of life and property. To avoid it, the task of precipitation nowcasting is imminent. To solve the problem, increasingly deep learning methods are proposed to forecast future radar echo images and then the predicted maps have converted the distribution of rainfall. The prevailing spatiotemporal sequence prediction methods apply ConvRNN structure which combines the Convolution and Recurrent neural network. Although improvements based on ConvRNN achieve remarkable success, these methods ignore capturing both local and global spatial features simultaneously, which degrades the nowcasting in the region of heavy rainfall. To address this issue, we proposed the Region Attention Block (RAB) and embed it into ConvRNN to enhance the forecast in the area with strong rainfall. Besides, the ConvRNN models are hard to memory longer history representations with limited parameters. Considering it, we propose Recall Attention Mechanism (RAM) to improve the prediction. By preserving longer temporal information, RAM contributes to the forecasting, especially in the middle rainfall intensity. The experiments show that the proposed model Region Attention Predictive Network (RAP-Net) has outperformed the state-of-art method.


Introduction
Precipitation nowcasting has vital influences in the field of transportation, agriculture, tourism industry and city alarming. Due to the higher spatial and temporal resolution of the radar echo image, it is suitable for forecasting the distribution of rainfall by predicting the future radar echo maps and converting each pixel to the rainfall intensity according to the Z-R relationship Shi et al. [2017]. Therefore, precipitation nowcasting is also defined as the spatiotemporal prediction problem.
Traditional approaches of precipitation nowcasting are motion field-based methods. The specific process can be briefly divided into three parts. First, the motion field is estimated by variational radar echo tracking methods such as optical flow . Second, the future radar reflectivities are advected by a semi-Lagrangian advection scheme under the assumption of stationary movement. Third, The performance of forecasts is evaluated by comparing to ground truth. However, these methods do not exploit abundant historical observation.
To overcome the limitation, some deep learning-based methods have been proposed to handle precipitation nowcasting Xingjian et al. [2015], Shi et al. [2017]. It builds a mapping from previous observation to future maps by learning from the abundant historical radar data. Generally, the prevailing approaches utilize the structure of ConvRNN which combines the Convolution Neural Network (CNN) and Recurrent Neural Network (RNN). Furthermore, to enhance the spatiotemporal representation ability, other types of neural networks such as Spatial Transformer Network (STN) Shi et al. [2017], Deformable Convolution Network (DCN) Wu et al. [2021] and Attention Module Lin et al. [2020] are introduced in the ConvRNN unit and obtain better performance.
However, existing ConvRNN models confront three drawbacks: 1) The convolution employed in the current input only extracts the local features instead of the large-scale representation due to fixed kernel size. It leads that useful information beyond the visual field of convolution dose not be captured and degrades the performance. 2) The convolution applied in the previous hidden state only transmits local previous representation to the current, which causes historical spatial information cannot make fully used. 3) The update process of temporal memory limits the long-term spatiotemporal representation preservation. Thus, the information involving high echo reflectivity is easily dropped. Although some remedial solutions Wang et al. [2018a] based on attention mechanism are proposed, they are not suitably applied in the large-scale inputs and long term prediction as to the limitation of the space occupation.
To address the first two problems, we propose the Region Attention Block (RAB) and embed it into the input and hidden state respectively. It exploits the global spatial representation and preserves the local feature simultaneously. RAB classifies each feature map into equal-sized tensors and the similar semantics gathered in the same tensor. Then, the attention module is executed to interact with the contents of all semantics. To this end, the large-scale feature map can be captured from the global view while maintaining local representation. Therefore, the larger spatial feature of the current input and previous hidden state can be preserved. Moreover, to capture the long-term spatiotemporal dependency of representation without increasing uncountable parameters, we present the Recall Attention Mechanism (RAM) to retrieval all historical inputs. More rainfall information is memorized thanks to this component. By adding these modules, the performance for heavy and middle rainfall has significantly improved. In brief, the main contributions of the paper are summarized as follows: 1. We first propose a new attention method, name Region Attention Block (RAB), to take capturing both global and local spatial features into account simultaneously to improve the spatial expressivity of feature maps.
2. We embed the RAB into the current inputs and previous hidden state to obtain the larger spatial information from the global view and persevere different semantics at the same time. For the same echo with large-scale size and long-range movement between the adjacent time, more useful spatial information can be extracted and predictions in those regions with heavy rainfall are more accurate.
3. We propose the Recall Attention Mechanism (RAM) to retrieval all historical inputs without limited parameters. The representation of middle rainfall intensity has maximized extend to be saved in the predicted unit.

Related Work
Traditional methods mainly focus on estimating the motion field between the adjacent radar maps and then the next prediction can be extrapolated based on this movement. Here, the motion field describes the direction and distance of each pixel that need to be moved at the next moment. To obtain the movement, Tracking Radar Echoes by Correlation (TREC) Wang et al. [2013] divides the whole radar maps into serval equal-sized boxes and calculates the motion vector of each pairs box center by searching the highest correlation between boxes at the adjacent time. Another type of approach is the optical flow-based method . It computers the motion field under pixel level based on the assumption that the brightness of pixels remains unchanged. Upon the idea, many algorithms Ryu et al. [2020] are represented to apply the radar maps with the large movement vector. However, the invariant brightness assumption is a conflict with the realistic movements of hydrometeors and massive historical data are utilized.
To overcome it, many deep learning-based methods are proposed to predict the radar sequence without the above unreasonable assumption. The prevalent methods apply the structure of ConvRNN. It combines Convolution Neural Network (CNN) and Recurrent Neural Network (RNN) to preserve the spatiotemporal feature of the historical sequence. Furthermore, Wang et.al added a spatial memory in predicted unit Wang et al. [2017Wang et al. [ , 2018b and attention mechanism in temporal memory Wang et al. [2018a] to enhance the spatiotemporal representation ability of short-term and longterm, respectively. Although these methods have remarkable performance, the visualization of predictions is usually burry due to the loss function and the architecture of the model Shouno [2020]. To handle the issue, Generative Adversarial Network (GAN) Xie et al. [2020] has been introduced in the ConvRNN model to improve predictive clarity. Nevertheless, the non-convergence and collapse problem still cause a negative influence on prediction. Moreover, ConvRNN models as the generator still play a decisive role in the framework of GAN.

Preliminary
The attention function can be described as a mapping from three vectors, name query Q ∈ R B×L×Dq , key K ∈ R B×L×D k and value V B×L×Dv , to output as the following formula: where d k is the dimension of key K and f commonly uses the dot-product(multiplicative). Attention module is also widely applied in the video prediction Wang et al. [2018a], Lin et al. [2020]. Specifically, as for a feature map F i ∈ R B×C×H×W , these three matrixes of query Q, key K and value V are generated by three different convolutions and keep the same shape with 4 Proposed Method

Problem Definition
The precipitation nowcasting task can be defined as the spatiotemporal sequence prediction problem Shi et al. [2017]. Based on historical observations X 0:t , it aims to forecast the future radar echo imagesX t+1:T that have maximum probability with ground truth X t+1:T as following: In this paper, t and T are set to 5 and 15 respectively, which means that ten continuous radar maps need to be predicted according to five historical images. [ht]

Overall Architecture
The interal structure of Region Attention Unit (RAP-Unit) is shown in Figure 1. The inputs include the current input X l t , previous hidden state H l t−1 , temporal memory C l t−1 , spatial memory M l t−1 and long-term historical representation is all previous inputs X 0:t at the bottom layer. The outputs of RAP-Unit are the current hidden state H l t , spatial memory M l t , temporal memory C l t and new long-term representation X l h . The details of calculation are presented according to following formulas: where ' * ' and '•' denote the convolution and Hadamard product respectively. The overall architecture of the proposed model RAP-Net is represented as Figure 2 shown. It refers the structure of PredRNN Wang et al. [2017] and generates the predictions from timestamp 2 to T . The red and blue arrows denote the delivering direction of spatial and temporal memory respectively. Different from the PredRNN, RAP-Net has another data flow to transmit long-term spatiotemporal information X l h . Besides, we notice that the majority of ConvRNN models employ this architecture. Therefore, the experimental result can be reflected the effectiveness of the predicted unit better.

Region Attention Blcok
To capture the large-scale spatial feature maps, the spatial attention module is applied in the video prediction task Lin et al. [2020]. It calculates the similarity matrix by multiplying elements from different locations as Figure 3(a) shown. However, over-fine grained operation between the elements often breaks the local representation. In order to preserve the whole local information, Vision Transformer (ViT) Dosovitskiy et al. [2020] directly divides the whole feature map into different patches and computes the similarity matrix between these regions in Figure 3(b). To this end, the local representation in different patches can interact with each other by attention operation. Therefore, it considers global and local information at the same time. However, the position of each patch is fixed in ViT model, which might cause the same object to be scattered in several neighbor patches and the local representation is destroyed to some extend.
To address the issue, we expect that these patches can be divided adaptively and those elements with similar semantic relationships are classified into the same patch shown in Figure 3(c). To realize this idea, we propose the Region Attention Block (RAB) whose structure is illustrated in Figure 4. First, a convolution and softmax layer are employed in the input feature map F i ∈ R B×C×H×W to generate F c ∈ R B×N ×H×W for distinguishing N classifications. Second, the orginal input F i is split N groups of feature maps F b ∈ R N ×B×C×H×W by the Split Module following this formula: where the P and Q indicate F i and F c , respectively. These groups denote various semantic information extract from different position. Third, F qk ∈ R B×N ×c×h×w is convolved by F b to furthermore exploit the feature of F b and reduce the parameters, where the c, h and w are smaller than C, H and W . Besides, F v ∈ R N ×B×C×H×W are outputed by a convolution layer applied in F b to preserve original information. Forth, three different convolutions are used to generate query Q, key K and value V based on F qk and F v . After flattening, Q ∈ R B×N ×c * h * w , K ∈ R B×N ×c * h * w and V ∈ R B×N ×C * H * W are fed into the attention function to obtain F a ∈ R B×N ×C×H×W that have been interacted Here, F a and F c are represented by P and Q, respectively. The result F a has the same size of input feature map F i . Finally, the structure of ResNet He et al. [2016] is introduced to achieve the final result F o ∈ R B×N ×C×H×W . In summary, the calculation process is described by the following formulas:

Recall Attention Mechanism
To capture the temporal long-dependencies of representation, Wang et.al Wang et al. [2018a] embedded the spatial attention module in the updating of temporal memory. However, it has two limitations 1): it saves abundant history temporal memories, which leads that the number of parameters easily exceeds the space occupancy as lead time goes.
2) The temporal memory has lost some information during the generation of various gates. Therefore, the preserved previous representation is not the whole information and long-term spatiotemporal expressivity is limited.
To address these issues, we propose the Recall Attention Mechanism (RAM) to enhance the long-term spatiotemporal representation ability with fixed space occupation as Figure 5 shown. First, we build an empty long-memory feature  map X 0 h ∈ R B×T ×C×H×W in the bottom layer and feed into the current input X t continually. The X 0 h thus contains all original previous inputs X 0:t ∈ R B×C×H×W . Secondly, a layer convolution neural network is employed to extract the feature of X 0 h and output the long-memory hidden state X 1 h ∈ R B×T ×C×H×W . Finally, X 1 h and the output H t of RAP-Cell feed into the channel attention module to generate new hidden states, where the X 1 h can be regarded as the value and key, and the H t represents the query. In this mechanism, the new output H t has recalled all original historical representation and long-term dependencies can be maximize preserved. Besides, the size of the long-memory feature map X h is limited at any time step. In the next layer, the input of the long-memory hidden state is the X 1 h from the bottom layer. To this end, the long-term historical representation can deliver to the next layer.

Dataset
The dataset is collected from the CIKM AnalytiCup2017 competition and covers the whole area of Shenzhen. For convenience, we name this public dataset to RadarCIKM. RadarCIKM has a training set and test set with 10,000 and 4,000 sequences, respectively. There are 2,000 sequences randomly sampled from the training set to build the validation set. Each sequence contains 15 continual observations within 90 minutes, where the spatial and temporal resolution of each map is 1km × 1km and six minutes, respectively. The range of each pixel is from 0 to 255 and each pixel p can be converted to radar reflectivity (dBZ) by Z-R relationship as follows:

Evaluation Metrics
In this paper, except for common measurements such as Structural Similarity (SSIM), Mean Absolute Error(MAE) in video prediction, we also applied the Heidke Skill Score (HSS) and Critical Success Index (CSI) which are used in precipitation nowcasting task. The HSS and CSI evaluate the fraction of correct forecasts after eliminating random predictions and the number of correct forecasts divided by the total number of occasions when the rainfall events were forecasted or observed, respectively. Specifically, the prediction and ground truth are converted to binary matric based on a threshold τ . Once the value of dBZ is larger than τ , it is set to 1 otherwise to 0. Next, the number of the True-Positive (TP, prediction=1 and truth = 1), False-Negative (FN, prediction = 0 and truth=1), False-Positive (FP, prediction=1 and truth=0) and True-Negative (TN, prediction=0 and truth=0) are counted. Finally, the HSS and CSI can be calculated by the following formulas:

Parameters Setting
The proposed RAP-Net takes five previous radar echo maps as inputs and outputs ten predictions. It utilizes four layers RAP-Units as shown in Figure 2, where the number of patches is set to 64. The Adam optimizer is applied to train our model with a 0.0001 learning rate. Besides, the early stopping and scheduled sampling strategies are also used to optimize our model. The loss function combines the L1 and L2 to train SST-LSTM. All experiments are implemented in Pytorch and conducted on NVIDIA 3090 Ti GPUs. The detail is available from the source code: https://github.com/luochuyao/RAP-Net.

Result and Analysis
Tables 1 and 2 show the evaluations of all models. We find that the RAP-Net achieves the smallest error and the highest structural similarity according to the MAE and SSIM. It shows that our model outperforms other models in terms of the whole prediction. Besides, the proposed model has significant superiority especially for the nowcasting in heavy rainfall regions. Because the HSS and CSI keep the top position in the middle and high thresholds (20 dBZ and 40 dBZ). For the state-of-art method, PFST-LSTM Luo et al. [2020], all measurements of it are exceeded by RAP-Net, which shows the performance of our model furthermore. Comparing with PredRNN, PredRNN++ and RAP-Net, we can see that they have similar SSIM due to applying the same architecture. However, the other evaluate indexes of RAP-Net is the highest, which implies the advance of RAP-Unit. Finally, we notice the SA-ConvLSTM Lin et al. [2020] have the best HSS and CSI in the lowest threshold (5 dBZ) and bad performance in the highest performance, which means the Region Attention can improve the prediction in the area with high radar echo compared to traditional attention mechanism.  Figure 6 shows an example of predictions from these models. The various colors denote the different ranges of reflectivity according to the color bar in the bottom of Figure 6. From the ground truth in the first row, the rainfall event is obviously the trend of increasing the rainfall intensity. However, only our model can forecast this trend and keep the intensity of the regions. The RAP-Net can generate a high reflectivity area, which can also explain why our model can achieve the highest evaluate index HSS and CSI in the middle and high thresholds.

Ablation Study
To investigate the influence of various modules, we conduct an ablation study to discuss the function of Region Attention Block to the current input and the last hidden state. The result of evaluations is shown in Tables 3 and 4. RAP-Cell x and RAP-Cell h denote the PredRNN model embedding the RAB into the input and hidden state, respectively. RAP-Cell model is the combination of RAP-Cell x and RAP-Cell h , and can also be regarded as RAP-Net without RAM. The various measurements of RAP-Cell x and RAP-Cell h are higher than PredRNN, which shows the validation of introducing Region Attention Block. Besides, the HSS and CSI of RAP-Cell have significant improvements in the highest threshold, which implies that RAB simultaneously employed in the input and hidden state contributes to the prediction in the heavy rainfall regions. Moreover, By comparing the RAP-Cell and RAP-Net, we notice that the RAM can raise the accuracy of nowcasting especially in the area with middle-intensity rainfall.  We also show predictions of the same sample in Figure 7. we find that RAP-Cell can generate the red area which is reflected by better evaluation indexes of HSS and CSI in the highest threshold. However, all forecasts except for RAP-Net have a gap in the radar echo block, which is obviously different from the ground truth. The improvement of prediction in middle rainfall intensity can be owed to the embedding of RAM.

Conclusion
In this paper, we propose the RAP-Net to handle the precipitation nowcasting task. On the one hand, It embeds the Region Attention Block to enhance the local and global spatial representation ability simultaneously as extracting and delivering the features in ConvRNN. The improvement significantly enhances the accuracy especially in those regions with heavy rainfall. On the other hand, we introduce the Recall Attention Mechanism to improve the temporal expressivity in the long term. It can preserve and retrieve longer historical information and effectively enhance the performance of prediction, particularly for the middle rainfall intensity. In the future, we will explore how to predict longer radar echo maps.