{'fraction_non_alphanumeric': 0.0320875338087042, 'fraction_numerical': 0.010982706335546266, 'mean_word_length': 4.661948955916474, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
FDNet: A Deep Learning Approach with Two Parallel Cross Encoding Pathways for Precipitation Nowcasting

With the goal of predicting the future rainfall intensity in a local region over a relatively short period time, precipitation nowcasting has been a long-time scientific challenge with great social and economic impact. The radar echo extrapolation approaches for precipitation nowcasting take radar echo images as input, aiming to generate future radar echo images by learning from the historical images. To effectively handle complex and high non-stationary evolution of radar echoes, we propose to decompose the movement into optical flow field motion and morphologic deformation. Following this idea, we introduce Flow-Deformation Network (FDNet), a neural network that models flow and deformation in two parallel cross pathways. The flow encoder captures the optical flow field motion between consecutive images and the deformation encoder distinguishes the change of shape from the translational motion of radar echoes. We evaluate the proposed network architecture on two real-world radar echo datasets. Our model achieves state-of-the-art prediction results compared with recent approaches. To the best of our knowledge, this is the first network architecture with flow and deformation separation to model the evolution of radar echoes for precipitation nowcasting. We believe that the general idea of this work could not only inspire much more effective approaches but also be applied to other similar spatiotemporal prediction tasks


Introduction
Precipitation nowcasting is the task of providing precise and timely prediction of rainfall intensity in a local region over a very short range (e.g., 0-6 hours) based on radar echo maps, rain gauge, and other observation data. Accurate prediction plays a vital role in our daily life as well as many public safety scenarios such as road condition alarms and flight schedules. Since the accuracy and timeliness are highly desired, compared with other traditional forecasting tasks like weekly average temperature prediction, precipitation nowcasting has become a more challenging mission in the field of weather forecasting. Of particular note is the forecasting of heavier rainfall, which occurs less often but has a higher real-world impact. Forecasting has become extremely difficult because of the low frequency of heavier rainfall events. Existing methods for precipitation nowcasting include numerical weather prediction (NWP) based methods and radar echo extrapolation based methods [1]. NWP is one of the most successful approaches to conducting medium-and long-range (up to 6 days) weather prediction [1][2][3]. The core of NWP is the complex and meticulous simulation of the physical equations in the atmosphere model. These physical equations are coincident with our current beliefs about the dynamical behaviors of the atmosphere. The NWPbased approach has high computing complexity and its accuracy strongly relies on the initial conditions, which could make them lose competitiveness to the extrapolation-based methods in a short period forecast.
Recently, radar echo extrapolation-based methods have been noticed and widely adopted [2,4] these years.
These algorithms rely on the extrapolation of observations by ground-based radars via optical flow techniques or neural network models. Optical flow-based [4][5][6] methods, as typical extrapolation-based methods, have drawn increasingly more attention, owing to their fast speeds and high accuracies. The approach is conducted in two stages by the extrapolation of radar observations.
Firstly, the wind is estimated by comparing two or more precipitation fields as seen by radar based on the optical flow estimation algorithms developed in computer vision. Secondly, the precipitation field is moved along the estimated directions of the wind. However, the success of these optical flow-based methods is limited because the flow estimation step and the radar echo extrapolation step are separated and it is no easy to determine the most appropriate model parameters.
Neural network models consider precipitation nowcasting as a real application of spatiotemporal predictive learning which generate images conditioned on given consecutive frames and have shown their advantages [7][8][9] in many real-world datasets. In spa-tiotemporal predictive learning, there are two crucial aspects: spatial correlations and temporal dynamics, and previous studies mainly focus on how to model these two sides in a unified architecture. The Convolutional LSTM (ConvLSTM) [7] architecture naturally considers these two aspects jointly by plugging the convolutional operations in recurrent connections. Predictive RNN (PredRNN) [9] extends ConvLSTM and in- In general, the evolution of radar echoes can be resolved into optical flow field motion and morphologic deformation. As shown in Fig.1  The rest of the paper is organized as follows. A brief review of related work is given in Section 2 and some preliminaries are introduced in Section 3. The detailed structure of the proposed network is described in Section 4. Implementation details and experimental results on challenging benchmarks are illustrated in Section 5.
The paper is concluded in Section 6.

Related Work
The precipitation nowcasting task can be formulated as a spatiotemporal prediction problem in which both the input and the prediction target are spatiotemporal sequences. Convolutional neural networks (CNNs) [11] and recurrent neural networks (RNNs) [12] have been widely used for learning spatial correlations and temporal dependencies from a spatiotemporal sequence. Naturally, CNNs are applied to extract spatial features. As for temporal dynamics, the existing architecture can be generally divided into two types: (1) CNNs-based models which treat the spatiotemporal sequence as multiple channels [13,14]  forts that extend the training data from 2D radar images to 3D images, while adapting the above ConvRNN models for the 3D-radar-extrapolation problem [24].
However, all those above work considers the motion and deformation together, which may lose sight of finegrained evolutions. To the best of our knowledge, only MCNet [10] decomposes the motion and content which independently captures the spatial layout and the corresponding temporal dynamics, but it does not consider the variation of the content such as accumulation, deformation, or dissipation. By contrast, our model is characterized to learn not only the tendency of global motion but also the variation of local deformation.

Structured Sequence Predictive Learning
Sequence predictive learning is the problem of predicting the most likely future length-K sequence given the previous J observations: x t+1 , ...,x t+K = argmax In this paper, we mainly focus on the regular gridstructured sequence predictive learning problem. x t can be represented by a tensor X ∈ R P ×M ×N . For precipitation nowcasting, x t is a 2D radar echo map.

Convolutional LSTM
Convolutional LSTM (ConvLSTM) [7] is a popular model for regular grid-structured sequences, which explicitly encodes the structured information into tensors by replacing the multiplications of dense matrices in classical LSTM with convolutions. The main equations of ConvLSTM are shown as follows: where σ(·) is the sigmoid activation function, * and denote the convolution operator and the Hadamard product, respectively.
The ConvLSTM determines the future state of a certain cell in the grid by the inputs and past states of its local neighbors, which is achieved by using a convolution operator in the state-to-state and input-tostate transitions. If the states are viewed as the hidden representations of moving objects, a ConvLSTM with a larger transitional kernel should be able to capture faster motions while one with a smaller kernel can capture slower motions [7]. ConvLSTM has been adopted as a building block in many complex structures [9,10,19] since its proposal. We also employ ConvLSTM as an elementary building block in our proposed FDNet architecture.

Model Architecture of FDNet
We take an end-to-end learning approach to predict future radar echo maps: given a dataset consisting of the previous J radar echo map sequences Finally, the combination & decoder takes the outputs from the deformation encoder and the shape encoder as inputs, and combines them to produce a pixel-level prediction of the next frame. The prediction of multiple frames can be achieved by recursively performing the above procedures over K time steps. We will describe the detailed configuration of the proposed architecture as follows.

Position Encoder and Shape Encoder
The position encoder extracts typical position features of pixels from a single frame in a sequence by where m t ∈ R w×h×c is the representation of the position feature of the current frame. f pos is implemented by a CNN including convolution layers and activation layers.
The shape encoder has the similar structure of the position encoder. It extracts important shape features from a single frame by s t = f shape (x t ), where s t ∈ R w ×h ×c is the representation of the shape feature of the current frame. f shape is also implemented by a CNN.
In order to examine whether the shape encoder and the position encoder can extract the characteristics of the shape and the position, Fig.3 visualizes the output feature maps from them, respectively. We can see that the model has learned to extract shape features and position features, and is working in the way as expected.
The shape encoder learns the fine shape details including pixel light and shade, while the position encoder captures coarse localization and outline of the image.

Flow Encoder
The flow encoder aims to capture the optical flow field motion between consecutive frames without considering the deformation of the content in the frames.
But how to capture the features of the flow effectively?
TrajGRU [8]    The shape encoder, deformation encoder and combination & decoder. Brown arrows carry morphologic deformation information. The input can be either the ground truth frame for input sequence, or the generated frame at previous time step. One frame is generated at each time step as the output.
ever, how would the network find this correlation?
We take a similar "corr" function as that in [25] to perform multiplicative patch comparisons between two feature maps. Let m t−1 ∈ R w×h×c , m t ∈ R w×h×c be multi-channel feature maps in time step t − 1 and time step t, respectively, and w, h and c are their width, height and the number of channels respectively. Given a maximum displacement d, for each location p x,y in the first feature map m t−1 , the corr function will compute the correlations between the feature of p x,y and every the second feature map m t . The "correlation" between point p x1,y1 in the first feature map and point q x2,y2 in the second feature map is defined as the dot product of feature vectors in point p x1,y1 and point q x2,y2 : Computing correlations among all patch combina- combinations, and thus we obtain a final output of size The output of the corr function will be fed into a ConvLSTM (see subsection 3.2) layer to encode the correlations. The advantage of such a structure is that we could learn the variation of optical flow by learning the parameters of the ConvLSTM.
After that, we take a one-hidden-layer convolutional neural network to extract the optical flow. TrajGRU [8] claims that motion patterns have different neighborhood sets for different locations and therefore the "optical flow" should be not just one but a set. In our experiments, we tested both single optical flow and multiple optical flows. But the results show that the single optical flow approach achieves a better performance.
As described in Fig.4(a), the flow encoder takes the position encoder m t−2 and m t−1 at time step t − 2 and time step t − 1 as inputs, respectively, and produces the hidden representations of optical flow at time step t, which is denoted by u t,l , v t,l . Fig.5(a) visualizes the "flow" information that the flow encoder outputs. We can see that the network has learned reasonable optical flow field motion information.

Deformation Encoder
The evolution of radar echo map sequence is a complex process. The shapes of radar echo maps may expand, contract, or change rapidly due to the complex atmospheric environment, which is quite different from that of other video sequences like moving digits [7] and KTH actions [9]. Therefore, it is necessary to distinguish the changes of shapes from the translational motions of radar echoes, and this idea inspires the design of the deformation encoder.
But what could be considered as the hidden representation of the deformation? In our study, the transformation of two adjacent radar echo maps is divided into two aspects: the optical flow fields movement and the shape deformation. Therefore the shape feature of the former image is conducted by a "warp" function same with that in [8]  U , V ∈ R H×W , we have: We denote the output of warp function on s t−1 as w t . Intuitively, w t can be viewed as the inferred shape feature at time step t, which is generated by transforming s t−1 using the predicted optical flow decoded u t,l , v t,l . We deliver w t and shape features of the image at time step t to a "diff" function, which is defined as element-wise subtraction. The output of the diff function will be fed into a stack of ConvLSTM layers to encode the deformation, denoted as d t+1 , as described in Fig.4(b). We use the hidden features out from the ConvLSTM layers rather than the diff func-tion as the encoder of the deformation, aiming to learn the evolution of it over time.
The output feature maps of the deformation encoder are visualized in Fig.5(b). It can be observed that the deformation encoder captures radar echo changes at different hierarchies. For example, the feature maps in orange boxes pay more attention to the darkest pixels, while the feature maps in red boxes are sensitive to the pixel changes in multiple scattered areas.

Combination & Decoder
The outputs from the above deformation encoder, w t and d t , encode a high-level representation of inferred image after optical flow field movement and shape deformation, respectively. Given these representations, the goal of the decoder is to generate a pixel-level prediction of the next framex t+1 ∈ R w×h×c . For this purpose, it first combines these two representations into a unified representation by w t ] ∈ R w ×h ×2c denotes the concatenation of d t and w t in the depth dimension, and c t ∈ R w ×h ×c denotes the combined high-level representation of optical fields motion. g comb is implemented by a CNN layer.
Then c t is delivered to the decoder, which places c t back into the original pixel space byx t+1 = g dec (c t ).
We employ the deconvolution network [28] for our decoder network g dec , which is composed of multiple successive operations of deconvolution, rectification, and convolution.

Dataset Description
We verify our model on two real-world radar echo datasets, HKO-7 [8] and SRAD2020 1 , collected by Hong Kong Observatory and Shenzhen Meteorological Bureau, respectively. Each dataset contains radar CAPPI reflectivity images recorded every 6 minutes, as detailed below.
• HKO-7 The HKO-7 dataset contains radar echo data from intensity of the data is as Table 1.  Note: x means the radar reflectivity factor.

Loss Function
We use an objective function composed of multiple weighted losses for all models. Given the training T . Our model is trained to minimize the prediction loss by Here, x andx are the target and predicted frames, respectively. λ pixel and λ gdl are hyperparameters that control the effect of each sub-loss during optimization.
L pixel is designed to guide the model to match the average pixel values directly, while L gdl is expected to guide the model to match the gradients of such pixel values, to alleviate the image blurring tendency of predicted frames. In our experiments, we set λ pixel = 1 and λ gdl = 1, which means that these two aspects have the same effect.
As shown in Fig.1 and Fig.2, the frequencies of different rainfall levels are highly imbalanced. It is known that heavier rainfalls always have much greater social impacts. Since it has been proved that training with balanced loss function is essential for good nowcasting performance of heavier rainfall [8], we use the weighted loss function in the sub-loss L pixel . Therefore we get where y, z ∈ R T ×W ×H are two T -length frame sequences, y k,i,j is the (i, j)th pixel value in the kth frame in y, and z k,i,j is the (i, j)th pixel value in the kth frame in z. w k,i,j is the weight corresponding to the (i, j)th pixel in the kth frame, relating to its rainfall intensity or radar reflectivity value.
In our experiments, we set p=1, 2, and thus we get As for the HKO-7 dataset, if we define r as the corresponding rainfall intensity of the pixel x, we set the weight of pixel x as: 2, 2 ≤ r ≤ 5; 5, 5 ≤ r ≤ 10; 10, 10 ≤ r ≤ 30; 30, r ≥ 30. (1) As for the SRAD2020 dataset, we set the weight of the pixel x as: (2) However, both L 1 loss and L 2 loss produce blurry predictions, increasingly worse when predicting further in the future. To sharpen the image prediction, we take a gradient difference loss (GDL) [13] as a mitigation strategy. The GDL loss L gdl (y, z) is defined as: where λ ≥ 1 and λ is an integer. Considering the training time, we only set λ = 1, which means that we only take the differencing between adjacent pixels.
We also conduct experiments without GDL loss on the SRAD2020 dataset, and the results show that the GDL loss improves the CSI score and the HSS score by 0.007 for all models on average.

Optimizer and Learning Rate
We train all compared models using Pytorch and optimize them to convergence using ADAM [29]. For ConvLSTM, TrajGRU, PredRNN and MIM, we set the initial learning rate to 0.001, and for FDNet, we set the initial learning rate to 0.0001. We use gradient clipping with clipping value 50 for these models. As Su [22] reports, we find that the Conv-TT-LSTM model is unstable at a high learning rate such as 0.001, but learns poorly at a low learning rate 0.0001. Therefore we use gradient clipping with learning rate 0.001 and clipping value 1 following [22].
We apply the scheduled sampling strategy [30] to all of the models, to gently change the training process from a fully guided scheme using the true previous data, towards a less guided scheme which mostly uses the generated data instead.

Parameter Initialization
All convolutional kernels are initialized by Xavier's normalized initializer [31], and the initial hidden/cell states in ConvLSTM, ST-LSTM and Conv-TT-LSTM are initialized as zeros.

Experimental Results
To evaluate the performance of our model, we measure the balanced mean square error (BMSE) and balanced mean average error (BMAE) [8]. BMSE is defined as: and BMAE is defined as: where N is the total number of frames and w(·) is the weight defined in Equation (1) and (2).
We also calculate the critical success index (CSI) and Heidke skill score (HSS) for multiple thresholds that correspond to different rainfall levels.
We choose to use the thresholds 20 dBZ, 30 dBZ, 40 dBZ and 50 dBZ. The CSI score is defined as: T P +F N +F P and the HSS score is defined as HSS = where T P means true positive, F P means false positive, T N means true negative and F N means false negative. A higher CSI score or a higher HSS score indicates a better prediction result. Table 4 shows the comparisons of different approaches for BMSE score of the average of all 20 time steps (up to 2 hours), 30 minutes, 60 minutes, 90 minutes and 2 hours ahead precipitation on both datasets. We can observe that FDNet outperforms the compared models on the average BMSE of 20-time-step predictions. Besides, though FDNet is not the best one for short-term prediction (30 minutes), it shows a stable superiority for long-term predictions. Table 5 shows the comparisons of different methods under CSI, HSS, BMSE and BMAE on the SRAD2020 dataset. It is worth noting that the CSI and HSS scores for thresholds 40 dBZ and 50 dBZ reflect whether the model's prediction of heavy precipitation is accurate and therefore it is often more concerned by meteorological experts. FDNet performs the best under CSI and HSS score for thresholds 30 dBZ, 40 dBZ and 50 dBZ.
We also give a frame-wise comparison of HSS score under thresholds 30 dBZ, 40 dBZ and 50 dBZ in Fig. 6.
HSS-40 and HSS-50 indicate the probabilities of severe weather conditions. FDNet performs better than baseline methods in most cases (except for MIM under HSS-50 at time steps 4 to 9).

Ablation studies
We perform four ablation studies to analyze the respective contribution of each module in our model.  Table 6 show that without separating encoder the model performs very poorly.  We also test the effect of the number of ConvLSTM  1 Note: each cell contains the mean score of the 20 predicted frames. '↑' means that the higher the score, the better. '↓' means that the lower the score, the better. 'x ≥ τ ' means the skill score at the τ dBZ. We mark the best result within a specific setting with bold face. layers in the deformation encoder. We evaluate models with 0/1/2/3 ConvLSTM layer(s) as shown in Table 6 group ( Fig. 4(b)); 2) only deliv- flow" means the number of ConvLSTM layers in the flow encoder; "no.defor" means the number of ConvLSTM layers in the deformation encoder, and "" in "no.defor" means that we reduce the deformation modeling branch; "out.flow" denotes if the output from the flow modeling branch is delivered to the decoder, and "out.defor" denotes if the output from the deformation modeling branch is delivered to the decoder.
ering the output from the flow modeling branch (w t in Fig. 4(b)) to the decoder; 3) feeding the concatenation of the output from the deformation modeling branch and the shape encoder to the decoder (concatenation of d t+1 and s t in Fig. 4(b)). The difference between setting (1) and setting (2) is that the latter still retains the deformation modeling information to iterate the prediction of the next step. The corresponding results are shown in Table 6

Results Visualization
We visualize four typical sequences of predicted radar echo map in Fig.7, Fig.8, Fig.9 and Fig.10, respectively. The first line shows the original radar echo map sequence. We compare the predicted results of our model with five baseline methods. In order to see how our proposed two pathways work, we also give the visualizations of partial predictions labelled with "flow branch restruction" and "deformation branch restruction". This is done by delivering the output from the flow encoder(w t in Fig. 4(b)) to the decoder and the concatenation of the output from the deformation encoder and the shape encoder to the decoder (concatenation of d t+1 and s t in Fig. 4(b)), respectively. All compared models take 21 historical radar echo images as inputs, and predict the next 20 images (radar echo maps for the next two hours).
It can be seen that the flow encoder mainly focuses on the movement of radar echoes as a whole, and can remember the general outline and scope. It helps the model avoid extreme expansions of radar echoes in prediction which appear in other methods (see Fig.7 and Fig.8). The deformation encoder is very sensitive to pixel changes, even the information lost during 'warp' procedure. Therefore it helps the proposed model remember the heavy rainfall information for a long time (see Fig.8 and Fig.9), while other models all lose this information step by step when modeling motion and transformation together. The deformation branch also equips the model with capability of predicting new coming rainfall (see Fig.10). Overall, benefiting from these two separate encoding pathways, FDNet's results are not only sharp enough but also more deterministic in future predictions.

Conclusions
In this paper, we have proposed a Flow-Deformation      The deformation modeling is particularly sensitive to newly generated rainfall. For example, the restruction result from the deformation branch successfully predicts the new coming heavy rainfall at the left bottom, which makes FDNet's predictions much closer to the ground truth numerically than other models.