{'fraction_non_alphanumeric': 0.026338371013782114, 'fraction_numerical': 0.011242841882212843, 'mean_word_length': 5.16474144445644, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A review of radar-based nowcasting of precipitation and applicable machine learning techniques

A 'nowcast' is a type of weather forecast which makes predictions in the very short term, typically less than two hours - a period in which traditional numerical weather prediction can be limited. This type of weather prediction has important applications for commercial aviation; public and outdoor events; and the construction industry, power utilities, and ground transportation services that conduct much of their work outdoors. Importantly, one of the key needs for nowcasting systems is in the provision of accurate warnings of adverse weather events, such as heavy rain and flooding, for the protection of life and property in such situations. Typical nowcasting approaches are based on simple extrapolation models applied to observations, primarily rainfall radar. In this paper we review existing techniques to radar-based nowcasting from environmental sciences, as well as the statistical approaches that are applicable from the field of machine learning. Nowcasting continues to be an important component of operational systems and we believe new advances are possible with new partnerships between the environmental science and machine learning communities.


Introduction
Heavy rainfall events can cause major disruption to human activities. It is desirable to predict these events ahead of time so that decision makers can take action to protect life, property and prosperity. Nowcasting, or short-term forecasting from observations, remains an important tool in predicting these events.
The essential goals of nowcasting are identical to those of all weather forecasting, with the only difference being the spatial and temporal scales involved. The World Meteorological Organization (WMO, 2016) distinguishes among the various forecasting time horizons as: "Usually forecasts for the next 0-2 hours are called nowcasting, from 2-12 hours very short-range forecasting (VSRF), and short-range forecasting beyond that; but the capabilities of the different ranges can vary upon variables and weather situations." Radar-based nowcasting emerged in an era of mainly synoptic and mesoscale weather prediction. Predicting rainfall during that time was a challenge for numerical weather prediction (NWP) models, since computational restrictions limited the resolution at which NWP models could operate. As a result, NWP models were able to capture mesoscale weather patterns such as fronts, but not the smaller-scale convective patterns that occur within mesoscale systems. Thus, these models had limited utility in predicting rainfall in the early hours of the forecast because of its dependence on the unrepresented small scales. To improve forecasts at early time steps, techniques based on extrapolating the current weather from radar or satellite observations -described as nowcasting -began to be developed. These techniques had the advantage of being able to represent rainfall at the same resolution as the observations, which was significantly higher than that used by the numerical models. This led to significant increases in forecast skill in the early hours of the forecast. The extrapolation models were based on advecting present rainfall observations along their current trajectories, and are described as advective nowcasting. Because these models were very simplistic compared to full NWP, their skill reduced far more quickly. Nonetheless, Browning (1980) estimated that extrapolation algorithms outperformed NWP models at forecast times up to around six hours, making them an essential tool in operational centers.
Since the 1980s, operational NWP performance has seen major technical improvement from two directions. First, model resolution has increased from 10-20 km grid cells to around 1 km, a scale the model can begin to resolve convective processes. Alongside this, improvements in data assimilation (DA) have made it possible to utilise high density radar and satellite observations, leading to more accurate model initialisation.
However, despite these advances, making optimal use of the information in high-resolution observations has proved a formidable challenge. In many ways convective scale NWP is still in its infancy, as many sources of predictability and simplifying balance relations that are present at synoptic scales no longer hold. A number of active research areas are summarised in Sun et al. (2014), and Yano et al. (2018). The upshot is that while NWP models undoubtedly add skill on synoptic timescales, this is not the case for the small spatial scales and short time frames relevant to nowcasting. Years of progress in NWP modelling have narrowed this gap, but have not eliminated it.
The simpler advective nowcasting models have also seen improvements over the last few decades, moving beyond purely advection-based models by incorporating diagnoses for potential areas of precipitation growth or decay. Such models still lag behind NWP in certain key respects: they lack the ability to model the full atmospheric state and associated interactions, and without a traditional DA step the self-consistency of their predictions cannot be guaranteed. Even so, this approach has the potential to extend the utility of observations-driven nowcasting further than advection alone.
In moving towards more general data-driven approaches, machine learning (ML) is a natural ally. Recent years have seen the ascendance of deep learning in particular, a family of ML algorithms which can resolve complex non-linear hierarchies of information. The flexibility and conceptual simplicity of ML make it a promising approach to utilising more diverse sources of information, as well as gleaning more complex effects, making observations-driven nowcasting a more powerful tool.
Realising this potential will be a significant research challenge. Several features of the nowcasting problem distinguish it from more conventional ML tasks: the need for dense pixel-wise prediction; the need to handle underlying structure at multiple spatial and temporal scales; the need to handle extremes and out-of-sample events; an underlying need for flow predictions and handling of regime changes; and the need to make and verify probabilistic predictions. For some of these challenges, such as dense prediction, the machine learning community has developed techniques that are promising solutions. For others, however, atmospheric science can help inform the next generation of ML approaches, which can be broadly applicable to other fields. Table 1 highlights key research areas, with pointers to the relevant sections. It also outlines areas of ongoing research, where more work is needed.
There is a demand for new tools that allow us to assimilate diverse spatial observations, and for decision-support tools that are fast, scalable and actionable; it is in these settings that we see the opportunity for ML. This review aims to bring the fields of atmospheric nowcasting and machine learning closer together, providing an overview of our current state of knowledge and future pathways in precipitation nowcasting. We begin by detailing the existing approaches to nowcasting using radar extrapolation, survey the approaches from machine learning that can be applied to the predictive problems in nowcasting, and conclude with a summary of some of the pathways for future research.

Persistence-based Nowcasting in Atmospheric Science
Much of the existing work in precipitation nowcasting attempts to incorporate knowledge of the physics of precipitation into simple models. Some of the best known traditional approaches to nowcasting precipitation, according to Germann and Zawadzki (2002), are centred around usage of various inductive biases, including: 1. climatological values: means, medians, modes, with variances representing uncertainty 2. Eulerian persistence, which predicts most recent observation for future ones 3. Lagrangian persistence, which assumes the state of each air parcel is constant, and all change is due to parcels moving with the background flow (advection) 4. persistence of convective cells and their properties The Eulerian persistence is described by the following equation (Germann and Zawadzki, 2002): where Ψ(·) is the observed precipitation field, t 0 is the forecast initial time, τ is the time difference, andΨ(·) is the forecast precipitation field. Meanwhile, the Lagrangian persistence adds into the equation the displacement vector λ (Germann and Zawadzki, 2002): The Lagrangian persistence assumption is generally the more applicable in short-term rainfall prediction, and is the basis for all current radar extrapolation models. In this section, we describe the structure and implementation of these models in their most basic form, before discussing extensions in sections 3 and 4.
Optical flow algorithms can be divided into two stages. The first is to estimate an advection field from two or more radar images. The second stage is to predict future observations using the estimated advection field. Within this framework, field advection algorithms differ in the implementation of one or both stages.

Advection field estimation
The main challenge in making use of the Lagrangian persistence assumption is estimating the displacement vector. The goal is, given a sequence of rainfall fields, to find a motion field (u, v) for which the following equation is satisfied: or equivalently, where Ψ is the radar reflectivity or a derived rainfall product (Horn and Schunck, 1981;Pierce et al., 2012). This equation is simply a restatement of the definition of Lagrangian persistence in the form of a differential equation.
A complication is that equation 4 is under-specified: many possible motion fields might satisfy the conservation equation, not all of which will be physically plausible. It is therefore necessary to impose extra conditions in order to make a useful estimation.
The simplest possible choice for constraining the advection field is to assume it is given by a single vector. Early work sought a single displacement vector which maximised the cross-correlation coefficient between sequential precipitation maps. This was done by searching a small set of perturbed vectors surrounding an initial guess, this guess being based on the centre of gravity of each image (Austin and Bellon, 1974), or a previous displacement (Austin, 1978).
While the simple assumption of uniform motion can work well for local analysis, it is unrealistic when considering larger areas. An initial attempt to allow for non-uniform motion was made by Rinehart and Garvey (1978), who divide the full area into a number of smaller blocks and maximise the cross-correlation for each block separately. However, as demonstrated in Tuttle and Foote (1990), this weaker constraint could lead to inconsistent motion estimates and discontinuous advection fields.
The problem of producing spatially consistent estimates of non-uniform motion can be solved by using a variational approach; that is, encoding additional desired properties of the advection field in a cost function to be minimised. This method is similar to one introduced to the optical flow literature in Horn and Schunck (1981). Here, the cost function to be minimised is: where J ψ is the residual from the conservation equation (3), Ψ the precipitation field,α the predicted advection, and J 2 is a smoothness penalty. The innovation is the smoothness term J 2 , which enforces a degree of spatial consistency. Various choices of J 2 are possible; Li et al. (1995) use the divergence ∂u ∂x + ∂v ∂y , while Bowler et al. (2004) instead use the Laplacian ∇ 2 V.
The variational problem can be solved using various optimisation algorithms, such as successive over-relaxation (SOR) (Li et al., 1995) and conjugate-gradient methods (Germann and Zawadzki, 2002). The full variational problem involves an expensive minimisation, and various tricks have been used to improve convergence. In Li et al. (1995) and Bowler et al. (2004), variational techniques are used as a form of post-processing for a solution found using a cheaper block-based method. Another option is to solve the full variational problem using a hierarchical approach, where solutions are computed at successively higher resolutions with each field being used as an initial guess for the next (Germann and Zawadzki, 2002;Zawadzki, 1994, 1995).

Numerical advection
Once an advection field has been found, it can be used to propagate radar observations forward in time. This is generally done using a semi-Lagrangian scheme. The basic idea is to divide the advection into short time windows and "follow the streamlines" of the advection field (Germann and Zawadzki, 2002). However, working with a discretised field is linked to the challenges of numerical modelling.
Semi-Lagrangian scheme The semi-Lagrangian scheme is a numerical integration method specifically designed for advection problems. Unlike generic solvers (e.g. Euler's method), it makes explicit use of the Lagrangian conservation constraint of equation (3). This states that the Lagrangian derivative is zero; that is, the measured quantity following an air parcel is constant: A natural interpretation is that the solution at time n uniquely determines the solution at time n + 1 on a grid which has moved with the flow. This interpretation forms the basis of a purely Lagrangian method of solution, as described in Wiin-Nielsen (1959). However, using an increasingly distorted grid is unworkable in practice. A solution to this, suggested in Sawyer (1963), is to repeatedly project the solution back onto the original grid using interpolation. This intermediate method is what is known as the semi-Lagrangian scheme.
Backward and forward schemes The most commonly used version of the semi-Lagrangian scheme is the backward one, which comprises the following steps (Diamantakis, 2013): 1. For each point on the target grid, find the corresponding departure point r d 2. Interpolate the previous solution to find the value at r d The first step is usually solved by fixed point iteration on the difference vector α = r − r d : that is, by iteratively calculating with 2 or 3 iterations normally considered sufficient (Germann and Zawadzki, 2002). For the second step, a range of interpolation methods are possible, with bilinear and bicubic interpolation being common choices (Bonaventura, 2004).
The forward version of the scheme is conceptually similar, except that it follows the air parcels forward in time instead of backward. Its basic steps are as follows (Bowler et al., 2004): 1. For each point on the source grid, find the corresponding destination point r 2. Apply a spreading kernel to distribute the influence of r onto the target grid The reason for replacing interpolation with a spreading kernel is that the destination points will not in general lie on a grid, and so standard methods such as bilinear interpolation do not apply. The problem then becomes selecting an appropriate kernel shape and lengthscale.
Numerical diffusion As discussed in Germann and Zawadzki (2002), both the forward and backward semi-Lagrangian schemes lead to the loss of small-scale features through numerical diffusion. In the backward scheme this is due to the interpolation step, while in the forward scheme it is due to the use of kernel spreading.
As mentioned above, there are various possibilities for the interpolation step in the backwards scheme. The simplest option of bilinear interpolation has the appealing property of making the scheme stable even for very long timesteps (Bonaventura, 2004). Unfortunately, this choice is known to lead to excessive numerical diffusion, meaning that high frequency features are lost in the advection. Cubic interpolation significantly reduces this diffusion (Bonaventura, 2004), and is widely used. However, Germann and Zawadzki (2002) found that even cubic interpolation distorted the high frequency spectrum too much for their purposes. This led them to consider a modified "interpolate once" approach: instead of interpolating the field at each timestep, each target grid point is traced all the way back to its original point of origin, where a single interpolation is carried out. They found that this modified method did indeed reduce artificial diffusion.
The forward scheme was chosen in Bowler et al. (2004) because, unlike the backward scheme, it is guaranteed to be conservative. However, they found that this conservation property produces a "banded" structure in areas of flow divergence. As noted in Germann and Zawadzki (2002), these numerical artefacts can be reduced or removed by increasing the kernel lengthscale, but at the expense of degrading small-scale features.

Cell-based advection
An alternative to the standard optical flow approaches for radar extrapolation nowcasting is given by cell-based (or object-oriented) methods. Unlike the full motion fields used in the algorithms above, cell-based methods focus on locating and tracking specific convective cells. This approach has an advantage when a specific severe storm is moving differently to surrounding storms, as it can be more easily identified and more accurately tracked.
Similar to field-based advection, cell-based methods consist of a flow estimation phase followed by an extrapolation phase. However, the way flow fields are estimated is quite different, consisting of an initial stage of cell identification followed by object tracking. Cells are typically identified by selecting radar regions above one (e.g. CONRAD), multiple (e.g. SCIT), or iteratively chosen sequences of thresholds (e.g. TRACE3D, TRT, TITAN). The advantage of multiple thresholds is that clusters of storm cells can be identified. The tracking procedure then involves extrapolating the position of cells found in a previous scan ('parent cell') to an estimated new position and checking if there is a match with cells found in the new scan ('children'). The matching is usually done by distance, but the CELLTRACK algorithm also accounts for 'shape similarity'.
Various methods differ in how they incorporate information from multiple previous time steps. For example, SCIT performs a least-squares fit, but TRACE3D takes a running weighted sum with the oldest information having the least weight. The original TITAN algorithm (Dixon and Weiner, 1993) uses an optimisation method for cell tracking.
Here all potential tracks are considered and a cost function based upon position difference and volume difference is minimised to select the most likely track. This method is based upon several assumptions, namely that cells move small distances between radar scans and retain their characteristics such as size and shape. Neither of these assumptions are satisfied even approximately and modifications have been made in Muñoz et al. (2018) to incorporate information from an optical flow based field tracker to constrain the solutions from the original method. The CELLTRACK algorithm also uses information from a field-based method (COTREC (Novak, 2007)) to provide a first guess for the new position estimates of cells during tracking.
Besides allowing for differential motion between nearby cells, cell-based approaches also enable the tracking and prediction of cell characteristics and life cycles. We will return to this idea in section 4.

Summary
Lagrangian persistence provides a foundation for most present-day nowcasting systems. Its use involves first estimating a motion field from a sequence of observations, then advecting the most recent observations along this motion field. The motion field is estimated using a variational minimisation or cell-tracking algorithm.
While Lagrangian persistence provides a useful foundation, the assumption is frequently flouted by real precipitation fields. The following sections will describe extensions to this simple model which loosen the unrealistic assumption of persistence.

Probabilistic and stochastic approaches
According to Bowler et al. (2006), the most significant sources of error for advection models are: 1. errors in estimating the advection field 2. neglect of temporal evolution of the advection field 3. neglect of Lagrangian evolution with the latter found to be the most significant source of model error.
There are two, non-exclusive, approaches to mitigate these sources of error. The first is to model the uncertainty due to these errors using a probabilistic approach. The second, applicable to the latter two points, is to attempt to improve the model by representing the neglected evolution processes. In this section we discuss probabilistic approaches, with evolution approaches being discussed in section 4.
In its most basic form, advective nowcasting is entirely deterministic. However, the importance of accounting for uncertainty of predictions is now widely recognised (Gneiting, 2008), especially in the case of precipitation, where forecast errors can have a significant effect on downstream applications (Vivoni et al., 2007). A number of approaches have therefore been proposed to account for the sources of error listed above, both probabilistic and stochastic.

Neighbourhood methods
A practical approach to estimating the uncertainty due to errors in the advection field is to consider the distribution of precipitation values in the area surrounding the target grid cell. In Andersson and Ivarsson (1991), the neighbourhood considered is a circle with radius proportional to the forecast time duration, from which precipitation values are drawn uniformly. Schmid et al. (2002) assume instead that advection errors follow a two-dimensional Gaussian distribution, with lengthscale proportional to the extrapolation time. Germann and Zawadzki (2004) provide a more detailed analysis of the optimal neighbourhood size as a function of time.
A related approach is taken by  and Xu et al. (2005), which frames the problem as a stochastic linear integro-difference equation (IDE) governed by a redistribution kernel k. The redistribution kernel plays a similar role to the neighbourhood, determining the motion and diffusion properties of the field. By allowing k to vary as a function of space, this model can achieve considerable flexibility. Inference, carried out using Markov chain Monte Carlo (MCMC), is correspondingly more intensive.

Stochastic methods (Scale decomposition)
The persistence time of precipitation features is known to be related to their spatial scale, with smaller features having shorter lifetimes (Germann and Zawadzki, 2002). It is sometimes seen as desirable to filter out these unpredictable scales in order to reduce the root-mean-squared forecast error (Seed, 2003;Turner and Zawadzki, 2004). An alternative to filtering unpredictable lengthscales is to model them stochastically, by injecting spatially-correlated noise.
Both filtering and stochastic simulation rely on an initial scale decomposition step. The simplest approach is to use the Fourier transform to move the field into spectral space, and then apply a Gaussian bandpass filter to divide it into frequency bands (Bowler et al., 2006;Seed, 2003). The full field can then be expressed as a sum of these bands: where K is the number of frequency levels. Since rainfall fields can be highly non-stationary, localised techniques such as the wavelet transform (Turner and Zawadzki, 2004) or short-space Fourier transform (Nerini et al., 2017) are sometimes preferred over standard Fourier methods.
When performing the spectral decomposition, it is standard to work with logarithmic radar reflectivity (dBZ) rather than the precipitation rate (Germann and Zawadzki, 2002), as rain rates are believed to be better represented by a multiplicative cascade model than an additive model, in a similar manner to turbulence (Schertzer and Lovejoy, 1987). The use of a logarithmic variable means that this can be rewritten as an additive process.
In the case of filtering, high-frequency components are simply truncated as they reach their limits of predictability (Seed, 2003). In stochastic methods, they are instead replaced by spatially correlated noise. This is done by convolving a Gaussian white noise field with a filter which captures the correlation properties. In earlier work such as Bowler et al. (2006), the filter is given by a theoretical power-law model; further details are given in Schertzer and Lovejoy (1987) and D' Apuzzo et al. (2008). An alternative, used more recently in Seed et al. (2013), is to use the Fourier transform of an observed rainfall field as a filter. Unlike the power-law method this approach is not limited to isotropic fields, and can be used to model noise with a strong directional element. Both approaches are limited to modelling stationary fields, and tend to produce unrealistic levels of noise in areas of high rainfall. However, the use of a logarithmic model does eliminate noise in dry areas.
In the STEPS system, both the precipitation field and the stochastic noise field are evolved using a hierarchy of secondorder auto-regressive (AR-2) processes (Seed et al., 2013). The coefficients of these processes determine the rate of Lagrangian evolution for a given spectral band.

Summary
In their most basic form, radar extrapolation approaches to nowcasting assume that the evolution of a precipitation field is primarily due to Lagrangian advection. Probabilistic methods relax this assumption by accounting for deviations from pure advection, either by interpreting the forecasts as neighbourhood rather than point predictions, or by removing the dependence on "unpredictable" scales by blurring them or replacing them with random noise.
The techniques described in this section do allow for some uncertainty in the development of the field due to Lagrangian evolution or variability in the advection field. However, these effects are limited to the inherent uncertainty within a given weather regime. They do not account for the more challenging problem of regime change, nor do they account for the possibility of predictable Lagrangian evolution. We will turn to these considerations in the following section.

Nowcasting convective development
One problem that has not been addressed by radar extrapolation techniques is that of predicting Lagrangian evolution, the largest remaining source of error in nowcasting. The absence is most notable in the case of convective initiation, development and decay. While these effects can be resolved by high-resolution NWP models, it is not straightforward to combine this predictive ability with the greater accuracy of radar extrapolation methods over short time periods. However, certain indicators have been found to improve nowcasting skill in handling more complex temporal evolution.
Most effort in predicting convection in nowcasting systems has been directed towards deep cumulus convection, because of its association with extreme rainfall. Three main approaches have been used in the literature to predict such events.
1. Combining advective nowcasting with an NWP model by blending or incorporating specific analyses.
2. An object-oriented approach, where deepening cumulus clouds are detected and classified according to their development potential. (Hand, 1996) 3. The detection of boundary layer convergence. (Mueller et al., 2003) A primary finding of the comparison study Wilson et al. (2004) was the significance of the third method. The authors state that "skill above extrapolation occurs when boundary layer convergence lines can be identified and utilized by a nowcasting system to nowcast storm evolution". The Auto-Nowcast system described in Mueller et al. (2003) uses radar to detect and track boundary layer convergence lines. Satellite data has also been used to detect boundary layer convergence, even in the absence of clouds (Roberts and Rutledge, 2003), as well as early cumulus growth. However, such signals will not lead to strong convection except in the presence of sufficient instability and moisture.
In response to this issue, several studies have investigated combining information on potential convection (nascent clouds or convergence) with information on whether the environmental conditions favour convective development. For example, advected satellite imagery can be used to identify potential convective cells, together with relevant NWP analysis fields such as convective available potential energy (CAPE) and convective inhibition (CIN) to predict further development, as in Steinheimer and Haiden (2007) and Mecikalski et al. (2015). The general framework of combining advection with diagnoses for initiation allows a great deal of flexibility in the choice of diagnosis, and various data sources and algorithms have been considered. Ahijevych et al. (2016) incorporates radar reflectivity as well as satellite data, while Han et al. (2017a) uses only 3-D radar analysis. Most of these works frame strong convection as a binary decision problem, to which they apply various statistical learning techniques such as logistic regression (Mecikalski et al., 2015), random forests (Ahijevych et al., 2016) and support vector machines (SVMs) (Han et al., 2017a). There is clearly scope for incorporating ML algorithms of arbitrary complexity within this framework, and we return to this theme in section 5.3.
In cell-based models, various methods can be used to extrapolate from the current characteristics of a cell to its future evolution. A simple example is the TITAN model (Dixon and Weiner, 1993), which assumes that growth and dissipation each follow a linear temporal trend. A more complex development model is given in Hand (1996) using a conceptual "flowchart" model of storm evolution. Mueller et al. (2003) combines a cell-based prediction scheme with additional predictors such as boundary layer convergence lines to more skilfully forecast storm evolution.
The Lagrangian evolution of convective systems is also closely connected to convective initiation. When the downdrafts of large convective cells reach the surface, they form an outward-spreading 'cold pool' of air. The cold pool can then act to lift the surrounding area, and in favourable conditions this leads to further secondary convection. In the case of multi-cell storms, secondary convection is what enables a storm to survive beyond the first hour or so (Bennett et al., 2006). It is also a significant cause of non-advective evolution. To capture these secondary effects, Hand (1996) use an object-based system in which mature cumulonimbus clouds initiate daughter cells in their expected downdraft location. This allows the system to predict some nonlinear evolution, although it does not address the initiation of first-generation cells.

Summary
Where it is possible at all, the prediction of Lagrangian evolution depends on incorporating additional diagnostics which give an indication of the potential for convective development. These include: CAPE and CIN; boundary layer convergence; orography; large-scale tropospheric subsidence; moisture; and cold pools due to cell downdrafts. In some cases, it may be possible to infer some of these diagnostics from observed precipitation fields. Others will require extra sources of information such as NWP analyses.

Machine learning approaches
While methods for radar extrapolation based on optical flow have seen considerable success, they have certain limitations due to assumptions of Lagrangian persistence and smooth motion fields. There have been efforts to relax these assumptions by incorporating specific mechanisms (such as convergence lines or cell life-cycles), or via probabilistic approaches; but as yet there has been no fully general solution short of a costly data assimilation cycle. Researchers have begun to explore machine learning techniques as a way to fill this gap. Machine learning provides an opportunity to capture complex non-linear spatio-temporal patterns and to combine heterogeneous data sources for use in prediction. In principle, it can enable us to weaken the assumption of Lagrangian persistence and produce more flexible models which take advantage of more varied sources of predictability.
Nonetheless, several challenges arise in adapting well-developed tools from deep learning to nowcasting. Having originally been developed to address quite different problems, even reproducing flow-based methods in a deep learning context can be challenging. Furthermore, the expressivity of deep learning models can produce results with undesirable properties such as blurred fields and missed extreme events if optimising with standard metrics such as mean-squared errors (MSE). Finally, interpretability remains a challenge. The main benefit of deep learning lies in its flexibility to learn arbitrary functions, but this very flexibility makes it more difficult to properly analyse the contribution of a model, and to understand which features are essential to its performance and which are incidental.
This section describes machine learning approaches that have already been applied to nowcasting, and also describes some techniques for related problems we believe could be fruitfully applied in this field.

Dense spatiotemporal prediction
In NWP there exists a well-developed paradigm for dense spatiotemporal prediction. In ML, the situation is still developing: there are numerous possible approaches, and not necessarily clear criteria for which approach is more appropriate for a particular task. This section describes three different options, based on recurrent convolution, flows, and direct prediction.

Spatiotemporal convolution
A common approach to temporal modelling is to use recurrent neural networks (RNNs) as they are designed to model an evolving state over time. This works by using the current network state as input into the next timestep. More recent variants such as long-short term memory (LSTM) (Hochreiter and Schmidhuber, 1997) and gated recurrent unit (GRU) (Cho et al., 2014) have incorporated gating structures to protect information over time, making it possible to learn long-range temporal dependencies. They have been applied to video prediction in Oh et al. (2015), which combines a convolutional encoder-decoder architecture with a fully-connected LSTM operating on the latent feature vector. This method was adapted by Shi et al. (2015) to retain spatial structure in the latent representations, enabling them to use convolution operations for LSTM state-state transitions instead of a fully connected layer.
Besides incorporating recurrence, the time dimension can also be handled as part of a convolutional architecture. One way to achieve this is by encoding the sequence of frames along the channel dimension and performing 2D-convolution (Mathieu et al., 2015). Another approach is 3D-convolutions, which are used by Vondrick et al. (2016) in the context of video generation. For prediction tasks, as future frames can depend only on past frames, a more suitable variant is causal convolutions (Oord et al., 2016a), which have been used for video prediction as part of a fully convolutional model in Xu et al. (2018).
At present, it is not clear whether combined convolutional-recurrent architectures or pure convolutional architectures are more suitable for spatiotemporal prediction. While there is evidence that well-designed convolutional architectures can outperform recurrent architectures for pure sequence forecasting (Bai et al., 2018), the comparison in Shi et al. (2017) suggests that this may not hold true in the spatiotemporal case. A detailed comparison of fully-convolutional and mixed recurrent-convolutional architectures for spatiotemporal prediction has yet to be carried out, and may well prove valuable.

Flows and deformations
Standard convolutional architectures learn a single set of filters that is then applied to every input. This is suboptimal for tasks such as nowcasting, because the appropriate transformation of the input is highly dependent on the input itself, and is also location-varying. Several works address this by using a two-part architecture, with one part predicting an appropriate set of filters and the other applying these filters to the input.
In Klein et al. (2015), this kind of two-part architecture was applied to a radar extrapolation task. Similar to Austin (1978), this model learns one or more filters that are applied uniformly to the input; that is, the deformation varies by input, but not by location. Later studies extended this architecture to non-stationary deformations, either by learning a spatially varying filter as in De Brabandere et al. (2016), or by learning several filters together with masks defining their domain of application, as in Finn et al. (2016). A similar non-stationary structure is used in Shi et al. (2017). However, in this case the model does not predict a convolutional filter, but instead predicts the recurrent connection structure, meaning it is less constrained by the convolution size.
As pointed out in Shi and Yeung (2018), these approaches are similar in spirit to predicting a spatial deformation field, and therefore comparable to the methods outlined in section 2. The work de Bezenac et al. (2019) on sea surface temperature prediction makes the use of flow fields more explicit. They use a network conditioned on previous timesteps to predict a motion field, which is then used as input to a warping scheme to transform the input field. The warping scheme computes a weighted average over a Gaussian centred at the previous location of each target cell; essentially a backwards semi-Lagrangian scheme described in section 2.2. This model is therefore closely connected to those discussed in section 2.1 that aimed to estimate a motion field by minimising Lagrangian evolution in the recent past. However, unlike those models there is no explicit minimisation of Lagrangian evolution, since the model is trained end-to-end based on the prediction accuracy. In principle, this gives the model greater flexibility in situations where Lagrangian evolution is not negligible.
More generally, there have been efforts towards training neural networks within a differential equation structure to model physical systems, without explicitly incorporating spatial flow. The authors of Ayed et al. (2019) demonstrate how this approach can be used to predict the future state of the shallow water model. Unlike the de Bezenac et al. (2019) model, this model has freedom to learn an appropriate warping scheme rather than just the motion field.

Direct prediction and training strategy
Besides recurrent and flow-based models, it is also possible to treat nowcasting as a direct image-to-image translation problem. This is the approach taken by Agrawal et al. (2019), who train a U-Net directly on one-hour prediction of radar fields.
In general, direct prediction may have an advantage over iterative prediction by allowing the model to tailor its predictions to the target forecast period. This is discussed in Shi and Yeung (2018), where it is argued that the choice of a training strategy is important for sequence forecasting as well as the choice of architecture. It concludes that direct prediction can lead to greater accuracy over short time periods, while iterative prediction may suffer from ac-cumulated errors due to the discrepancy between its training and test objectives. On the other hand, iterative models are easier to train and can be used to generate predictions at any distance into the future, while direct prediction is restricted to the time range it has been trained against. A potential synthesis can be found in the boosting strategy of Taieb and Hyndman (2014), which uses a core iteratively trained model with directly trained adjustments. The recent work of Sonderby et al. (2020) also combines the two strategies by conditioning an iterative model on the target lead time.

Sharp predictions and uncertainty
In machine learning, spatiotemporal sequence forecasting has most often been treated as a deterministic problem. As observed by Mathieu et al. (2015), this approach can have unwanted consequences if the underlying system is not deterministic. If the distribution is multi-modal, attempting to minimise the mean squared error loss will tend to average over the different modes, leading to blurry predictions.
The alternative is to use a generative approach, sampling from the distribution of possible futures instead of returning a single "best" prediction. In the spatiotemporal case, this distribution will generally be complex and high-dimensional, making it difficult to model. Even so, a number of methods have been proposed for generative modelling of spatiotemporal sequences. These methods belong to three main categories: adversarial training, sequential conditional pixel generation, and latent random variable models.

Adversarial training
As proposed in Goodfellow et al. (2014), generative adversarial networks (GANs) consist of a pair of networks which are trained in an alternating fashion. One of these networks (the generator) attempts to generate samples from the training distribution, and the other (the discriminator) tries to distinguish between the generated samples and training data. Adversarial training is equivalent to minimising the divergence between the generated and target distributions, and can be viewed as constructing a learned loss function which targets the task at hand, embodied in the discriminator. For prediction tasks, it is essential to be able to condition the generated output on previously observed frames; this can be achieved by providing the conditioning information as input to both the generator and the discriminator (Goodfellow et al., 2014).
Several studies have considered adversarial training for video generation (Denton et al., 2015;Vondrick et al., 2016) and video prediction (Jang et al., 2018;Mathieu et al., 2015). In particular, Mathieu et al. (2015) found that an adversarial training objective led to consistently sharper predictions than either the ℓ 2 or ℓ 1 norm.
Beyond producing sharp predictions, we may hope that the distribution of samples produced by a GAN is close to the true distribution. This is known to be the case in the idealised scenario of unlimited network capacity and training data, provided the model converges (Goodfellow et al., 2014). However, in more realistic scenarios convergence does not imply that the true distribution has been well approximated (Arora et al., 2017), and in practice GANs often suffer from mode collapse (Santurkar et al., 2017). Evaluating high dimensional generative models is a significant open problem (Gulrajani et al., 2019). These issues are especially acute for GANs because they do not explicitly model a probability distribution.

Sequential conditional pixel generation
Besides adversarial methods, other techniques aim to specify the full conditional distribution over predictions. One approach, described in Kalchbrenner et al. (2016), is to factorise the likelihood as the product of conditional probabilities over pixels, so that each pixel is conditioned on previous timesteps and on previously generated pixels in the current timestep (in their implementation, the pixels above and to the left). This conditioning ensures that the sampling of all pixels is consistent, in that they are all drawn from the same mode of the full distribution. The only requirement is that the model for the conditional distributions should be flexible enough to capture the dependencies (for example, it should not be assumed to be unimodal) while still being simple enough to provide tractable inference. In Kalchbrenner et al. (2016), the authors choose a discrete distribution over 256 potential values, essentially modelling the distribution by a histogram.
It is worth noting that Kalchbrenner et al. (2016) also model conditional dependencies across colour channels by imposing an arbitrary order. Although radar-based nowcasting is usually limited to a single channel, the same technique could be useful for modelling dependencies between meteorological variables (e.g. temperature and rainfall), or between radar and satellite observations.
In principle, the value of a pixel can be conditioned on all previous timesteps and pixels using a gated RNN, as described in Oord et al. (2016b). An alternative option is to condition on only a bounded region of pixels, which can be done using a CNN. This approach makes training significantly faster, since the convolutions can be run in parallel; the downside is that it introduces artificial conditional independence assumptions (Oord et al., 2016c). The video prediction model in Kalchbrenner et al. (2016) uses a hybrid approach, namely convolution in the spatial dimensions and a convolutional LSTM over time.
In either case, whether using the RNN or CNN version, using the model to generate samples requires generating every pixel sequentially. As a result, this class of models is time consuming for spatiotemporal data, especially at high resolutions.

Latent random variable models
Most deep learning research on latent random variable models focuses on variational autoencoders (VAEs). VAE is an encoder-decoder architecture which attempts to learn an embedding of the observed fields into a latent space. The model is trained by approximate inference, as detailed in Kingma and Welling (2014). The training objective is essentially to minimise the reconstructive error, but with an extra regularisation term which seeks to minimise the divergence between the encoded latent distribution and some fixed prior distribution. This prior is often taken to be a Gaussian with a diagonal covariance matrix, but this is not essential. What matters is that the prior distribution is known and can be sampled; by regularising the encoded random variables so that they are close to the prior, it is then possible to use samples from the prior to generate samples from the target distribution by passing them through the decoder.
Several works have incorporated a VAE component in their architecture to create a probabilistic model, such as for image prediction (Xue et al., 2016) and segmentation (Kohl et al., 2018). In the context of video prediction, both Babaeizadeh et al. (2017) and Denton and Fergus (2018) have proposed combining a VAE with a spatiotemporal LSTM predictor. However, the models differ in their treatment of the latent random variables: while Babaeizadeh et al. (2017) uses samples from a constant distribution to capture the system stochasticity, the model of Denton and Fergus (2018) learns to predict a distinct prior at every timestep. The time-varying prior is similar to that used by Chung et al. (2015), and was found to significantly improve the sharpness of predictions. It is also intuitively a better model of nonstationary environments, since the form of predictive uncertainty could change entirely from one frame to another.
A related group of generative models, designed to perform approximate inference, is flow-based models (NICE (Dinh et al., 2014); RealNVP (Dinh et al., 2016)). Instead of using a variational autoencoder to train the parameters of the latent distribution, these models use invertible neural networks, which enable direct optimisation of the log likelihood. The idea is to approximate complex, highly non-Gaussian posteriors using a series of invertible transformations; this technique is described in Rezende and Mohamed (2015) as variational inference with normalising flows. Flow-based models have been applied to conditional video prediction by Kumar et al. (2019). In this work, invertible networks are used to connect the observed and latent variables at each timestep and level of detail, while non-invertible networks are used to model the conditional dependencies between timesteps and levels.

Additional sources of predictability
Due to their flexibility as universal function approximators, machine learning methods could be expected to tease out additional sources of predictability implicit in the nowcasting problem, beyond that exploited by extrapolation. One example is that of accounting for uncertainty with blurred predictions. Others include: 1. influence of orography 2. life-cycle of convective cells 3. predictable motion field evolution 4. convergence lines 5. stable and unstable regimes.
Each is a potential source of predictability which could lead to improved skill compared to pure optical flow. In order to realise this additional skill, an ML model needs to have access to the right information. This could be provided to the model in the form of additional predictors, or in some cases extracted by the model itself from information already implicit in the radar data. For example, orographic fields could be used as input to an ML model, or the model could infer this information from raw location data. Likewise, convergence lines could be detected by adding assimilated wind fields as an input, but could also potentially be inferred from spatial patterns in the radar data using a convolutional neural network (CNN).
The example that has so far received most attention is incorporating orographic data. Foresti et al. (2019) found that a neural network trained on spatial location and flow direction had significantly better skill at predicting growth and decay of precipitation than a simple Eulerian persistence model, with inclusion of the previous rainfall value improving performance further. The effect was particularly strong over mountainous areas. Another example is found in Franch et al. (2020), where orographic data was used as input to a post-processing model stacked on top of a deep learning ensemble to improve predictions of extreme rainfall.
Motion field evolution is a promising target for ML because, unlike standard optical flow, most of the models discussed in sections 5.1.1 and 5.1.2 are able to represent time-varying motion fields. While not itself a machine learning model, Ryu et al. (2020) gives an encouraging signal that the evolution of motion fields may have some predictability without incorporating a full NWP model; they found that evolving the motion field based on Burgers' equation improved CSI compared to assuming a constant motion field.
As outlined in section 4, machine learning techniques have also been used to predict strong convective initiation by incorporating various analyses. Another attempt in this direction is Han et al. (2017b), in which a support vector machine (SVM) is trained to predict whether a box will contain high rainfall 30 minutes into the future based on vertical velocity and buoyancy values in nine neighbouring boxes. The input data is based on high-resolution VDRAS analysis (Sun and Crook, 1997). It is argued that the SVM can predict storm growth and convective initiation, although this may be an artefact due to the over-forecasting bias shown by the model.
So far, most research into sources of predictability has either treated the problem in isolation (as in the case of Foresti et al. (2019) or used the predictions to augment simple radar extrapolation. There has not yet been a study of how these sources of predictability are utilised by more complex deep learning architectures such as those described in sections 5.1 and 5.1.2. In part, this is no doubt due to interpretable machine learning (McGovern et al., 2019) being a challenging area in its own right. Perhaps more significantly, such process-oriented analysis requires an understanding of the mechanisms at play in meteorology and atmospheric science, as well as expertise in ML: further motivation for bridging the gap between the two fields.

Summary
There has been considerable progress in adapting deep learning methodology to dense spatiotemporal prediction tasks. Nowcasting itself has been addressed by direct prediction, as well as composite architectures combining recurrent and convolutional connections, and others using temporal convolutions. There have also been efforts to account for differing background flows by making the connection structure itself a predictand.
In other contexts, deep learning models have been used to predict flow fields that are used to transform the input, with gradients back-propagated through the flow. Although these methods have not been specifically applied to nowcasting there is a clear connection with the advection models discussed in section 2, and comparing these approaches could be an interesting direction for future research.
Another area with considerable scope for development is handling the multimodal system dynamics which occur in nowcasting. This will require using a generative approach, such as adversarial training, sequential pixel prediction, or variational auto-encoders. Each of these methods has been applied with considerable success in video prediction and generation; in particular, using a variational auto-encoder with a distinct prior predicted at each time-step seems a promising approach for atmospheric science (Chung et al., 2015).
A key advantage of machine learning models is their flexibility in combining data sources and utilising more varied sources of predictability. Despite this potential, so far little attention has been paid to combining multiple data sources, with the exception of orographic data which has been shown to hold predictive skill (Foresti et al., 2019). It also remains to be shown whether deep learning architectures such as CNNs are able to extract additional predictability from the spatial structure of input fields.

Discussion and Conclusion
The nowcasting problem poses several distinct research challenges: the need to make dense predictions; the need to handle underlying structure at multiple spatial and temporal scales; the need to handle extremes and out-of-sample events; an underlying need for flow prediction and handling of regime changes; and the need to make and verify probabilistic predictions. This review identifies some methods that may help solve those challenges.
Making the best possible use of other observational data to aid short-term rainfall prediction remains an active and challenging research area. There is opportunity to make progress in this area by fusing machine learning and meteorology; machine learning for its ability to extract meaning from high-dimensional data, and meteorology for its elucidation of the underlying physics driving evolution.
Incorporating physical mechanisms into ML models is a promising direction for further research, that may help to improve predictions and interpretability. This is especially true at longer time scales, where understanding the physics of the full atmosphere becomes important for understanding the evolution of the rainfall field. Physics-based advection models would be one example of this type of hybrid approach, as would modelling the effects of orography. Predicting convective initiation and evolution is a clear priority for future research in this area.
An advantage of deep learning, specifically CNNs, is their ability to extract latent information from spatial structure. This may enable them to make more efficient use of the data provided to them. For example, it may be possible to infer areas of developing boundary layer convergence from radar data within the model, rather than including it as an additional input. Such models may provide an alternative route to the incorporation of physical mechanisms discussed above; but there is still work to be done investigating the flow of information through these complex models.
One important development in the literature on extrapolation-based nowcasting has been the move away from optimal mean prediction and towards stochastic and probabilistic approaches, in order to account for the chaotic nature of convective evolution. This viewpoint also seems likely to be useful for approaches based on machine learning. The extensive literature on deep generative models gives hope that they will help to shed light on the challenging problem of representing uncertainty of convective-scale evolution, which is frequently multi-modal and nonlinear.
Finally, any new research will need to consider carefully the question of verification. A huge number of possible metrics are available, each of which embodies a particular view of what it means to be a "good" forecast. There is no general consensus on a single best verification metric which should be used, and such a consensus probably is not possible. The most reasonable approach seems to be to use a range, including at least one point-based metric, one spatial metric, and one for assessing ensemble calibration where applicable.
The weather touches every part of human life and with the need for greater resilience to changing environmental conditions the need for more accurate nowcasting is of increasing importance. There is much opportunity for new research and we believe that the fusion of the environmental sciences and machine learning holds much promise.