{'fraction_non_alphanumeric': 0.02893995011052749, 'fraction_numerical': 0.007564541970025756, 'mean_word_length': 4.899030984567532, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 2, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Shallow Learning for Fluid Flow Reconstruction with Limited Sensors and Limited Data

In many applications, it is important to reconstruct a fluid flow field, or some other high-dimensional state, from limited measurements and limited data. In this work, we propose a shallow neural network-based learning methodology for such fluid flow reconstruction. Our approach learns an end-to-end mapping between the sensor measurements and the high-dimensional fluid flow field, without any heavy preprocessing on the raw data. No prior knowledge is assumed to be available, and the estimation method is purely data-driven. We demonstrate the performance on three examples in fluid mechanics and oceanography, showing that this modern data-driven approach outperforms traditional modal approximation techniques which are commonly used for flow reconstruction. Not only does the proposed method show superior performance characteristics, it can also produce a comparable level of performance with traditional methods in the area, using significantly fewer sensors. Thus, the mathematical architecture is ideal for emerging global monitoring technologies where measurement data are often limited.


INTRODUCTION
The ability to reconstruct coherent flow features from limited observation can be critically enabling for applications across the physical and engineering sciences [12,63,14,50,74]. For example, efficient and accurate fluid flow estimation is critical for active flow control, and it may help to craft more fuel-efficient automobiles as well as high-efficiency turbines. The ability to reconstruct important fluid flow features from limited observation is also central in applications as diverse as cardiac bloodflow modeling and climate science [11]. All of these applications rely on estimating the structure of fluid flows based on limited sensor measurements.
More concretely, the objective is to estimate the flow field x ∈ R m from sensor measurements s ∈ R p , that is, to learn the relationship s → x. The restriction of limited sensors gives p m. The sensor measurements s are collected via a sampling process from the high-dimensional field x. We can describe this process as where H : R m → R p denotes a measurement operator. Now, the task of flow reconstruction requires the construction of an inverse model that produces the field x in response to the observations s, which we may describe as where G : R p → R m denotes a non-linear forward operator. However, the measurement operator H may be unknown or highly-nonlinear in practice. Hence, the problem is often ill-posed, and we cannot directly invert the measurement operator H to obtain the forward operator G. Fortunately, given a set of training examples {x i , s i }, we may learn a function F to approximate the forward operator G. Specifically, we aim to learn a function F : s → x which maps a limited number of measurements to the estimated state x: x = F (s) , so that the misfit is small, e.g., in a Euclidean sense over all sensor measurements F(s) − G(s) 2 2 < , where is a small positive number. Neural network based inversion is common practice in machine learning [52], dating back to the late 80's [75]. This powerful learning paradigm is also increasingly used for flow reconstruction, prediction, and simulations [46,69,42,17,32,70]. In particular, deep inverse transform learning is an emerging concept [56,41,1,73], which has been shown to outperform traditional methods in applications such as denoising, deconvolution, and super-resolution.
Here, we explore shallow neural networks (SNNs) to learn the input-to-output mapping between the sensor measurements and the flow field. Figure 1 shows a design sketch for the proposed framework for fluid flow reconstruction. We can express the network architecture, which we denote as shallow decoder (SD), more concisely as follows: s → first hidden layer → second hidden layer → output layer → x.
SNNs are considered to be networks with very few hidden layers. We favor shallow over deep architectures, because the simplicity of SNNs allows faster training, less tuning, and easier interpretation (and also since it works, and thus there is no need to consider deeper architectures).
There are several advantages of this mathematical approach over traditional scientific computing methods for fluid flow reconstruction [18,24,13,71,50]. First, the SD provides a supervised joint learning framework for the low-dimensional approximation space of the flow field and the map from the measurements to this low-dimensional space. This allows the approximation basis to be tailored not only to the state space but also to the associated measurements, preventing observability issues. In contrast, these two steps are disconnected in standard methods (discussed in more detail in Section 2). Second, the method allows for flexibility in the measurements, which do not necessarily have to be linearly related to the state, as in many standard methods. Finally, the shallow decoder network produces interpretable features of the dynamics, potentially improving on classical proper orthogonal decomposition (POD), also known as principal component analysis (PCA), low-rank features. For instance, Figure 2 shows that the basis learned via an SNN exhibits elements resembling physically consistent quantities, in contrast with alternative POD (PCA-based) modal approximation methods that enforce orthogonality.
Limitations of our approach are standard to data-driven methods, in that the training data should be as representative as possible of the system, in the sense that it should comprise samples drawn from the same statistical distribution as the testing data.
The paper is organized as follows. Sec. 2 discusses traditional modal approximations techniques. In Sec. 3, we briefly discuss shallow learning techniques for flow reconstruction. Then, in Sec. 4, Figure 1: Illustration of the shallow decoder which maps a few sensor measurements s ∈ R 5 to the estimated field x ∈ R 78,406 . In other words, this neural network based learning methodology provides an end-to-end mapping between the sensor measurements and the fluid flow field.  These dominant features show that the SD constructs a reasonable characterization of the flow behind a cylinder. Indeed, by not constraining the modes to be linear and orthogonal, as is enforced with POD, a potentially more interpretable feature space can be extracted from data. Such modes can be exploited for reconstruction of the state space from limited measurements and limited data.
the specific implementation and architecture of our shallow decoder is described. Results are presented in Sec. 5 for various applications of interest. We apply the shallow decoder to several prototypical flow field examples, considering both point-wise and sub-gridscale measurements. We aim to reconstruct (a) the vorticity field of a flow behind a cylinder from a handful sensors on the cylinder surface, (b) the mean sea surface temperature from weekly sea surface temperatures for the last 26 years, and (c) the velocity field of a turbulent isotropic flow. We show that a very small number of sensor measurements is indeed sufficient for flow reconstruction in these applications. Further, we show that the shallow decoder can handle non-linear measurements and is robust to measurement noise. The results show significantly improved performance compared to more traditional modal approximations techniques. The paper concludes in Sec. 6 with a discussion and outlook of the use of SNNs for more general flow field reconstructions.

BACKGROUND ON HIGH-DIMENSIONAL STATE ESTIMATION
The task of interpolating from a limited number of measurements to the high-dimensional statespace is made possible by the fact that the dynamics for many complex systems, or datasets, exhibit some sort of low-dimensional structure. This fact has been exploited for state estimation using (i) a tailored basis, such as POD, or (ii) a general basis in which the signal is sparse, e.g., typically a Fourier or wavelet basis will suffice. In the former, gappy POD methods [28] have been developed for principled interpolation strategies [18,24,13,71,50]. In the latter, compressive sensing methods [15,22,3] serve as a principled technique for reconstruction. Both techniques exploit the fact that there exists a basis in which the high-dimensional state vector has a sparse, or compressible, representation. In [51], a basis is learned such that it leads to a sparse approximation of the high-dimensional state while enforcing observability from the sensors.
Next, we describe standard techniques for the estimation of a state x from observations s, and we discuss observability issues. Established techniques for state reconstruction are based on the idea that a field x can be expressed in terms of a rank-k approximation where {φ j } j are the modes of the approximation and {ν j } j are the associated coefficients. The approximation space is derived from a given training set using unsupervised learning techniques. A typical approach to determine the approximation modes is POD [4,18,24,50]. Randomized methods for linear algebra enable the fast computation of such approximation modes [49,23,37,26,27,25]. Given the approximation modes Φ, estimating the state x reduces to determining the coefficients ν from the sensor measurements s using supervised techniques. These typically aim to find the minimum-energy or minimum-norm solution that is consistent in a least-squares sense with the measured data.

Standard approach: Estimation via POD based methods
Two POD-based methods are discussed, which we will refer to as pod and pod plus in the following. Both approaches reconstruct the state with POD modes, by estimating the coefficients from sensor information. The POD modes are obtained as the k most dominant left singular vectors of a training set X = (x 1 . . . x n ): where Φ ∈ R m×k denotes the left singular vectors and V the right singular vectors. The corresponding singular values are the diagonal elements of Σ.

Standard POD-based method
Let a linear measurement operator H : R m → R p describe the relationship between the field and the associated observations, s = H x. The approximation of the field x with the approximation modes {φ j } j is obtained by solving the following equation for ν: The standard approach is to simply solve the following least-squares problem The solution with the minimum L 2 -norm is given by: with the superscript + denoting the Moore-Penrose pseudo-inverse. In this situation, the highdimensional state is then estimated as This approach is hereafter simply referred to as POD.

Improved POD-based method
This above described approach requires explicit knowledge of the observation operator H and is subjected to ill-conditioning of the least-squares problem. These limitations render this "vanilla flavored" approach often impractical in many situations, and they motivate an alternative formulation.
The idea is to learn the map between coefficients and observations without explicitly referring to H. It can be implicitly described by a, possibly nonlinear, operator P : R k → R p typically determined by minimizing the Bayes risk, defined as the misfit in the L 2 -sense: where µ s,ν is the joint probability measure of the coefficients and the observations. We assume the training set is representative of the underlying system, in the sense that it should contain independent samples drawn from the stationary distribution of the physical system at hand. The Bayes risk is then approximated by an empirical estimate, and the operator P is determined as ( When the measurement operator H is linear, P is then an empirical estimate of H Φ, the contribution of the basis modes {φ j } j to the measurements s. Compared to the closed-form solution in Eq. (2), this formulation brings flexibility in the properties of the map P . For instance, regularization by sparsity can be enforced in P , via L 0 -or L 1 -penalization. Expressing Eq. (3) in matrix form yields: where S ∈ R p×n and N ∈ R k×n respectively refer to the training data measurements {s i } i and coefficients {ν i } i . It immediatly follows and the approximation obtained by pod plus is finally given by the solution to the following least-squares problem ν ∈ arg min ν s − P ν 2 2 .
However, ν ∈ R k is typically higher-dimensional than s ∈ R p , and thus the problem is ill-posed. We then make use of the popular Tikhonov regularization, selecting the solution with the minimum L 2 -norm. This results in a ridge regression problem formulated as: with λ > 0 the penalization parameter typically estimated through k-fold cross-validation. As will be seen in the examples below, penalization of the magnitude of the coefficients can significantly improve the performance of the POD approach.

Observability issue
The above techniques are standard in the scientific computing literature for flow reconstruction, but they bear a severe limitation. Indeed, since it is derived in an unsupervised fashion from the set of instances {x i } i , the approximation basis {φ j } j is agnostic to the measurements s. In other words, the approximation basis is determined with no supervision by the measurements. To illustrate the impact of this situation, let ν = Φ + x be the least-squares estimate of the approximation coefficients for a given field x. The difference between the least-square estimate coefficients ν and the coefficients ν obtained from the linear sensor measurements s writes and the error in the reconstructed field is obtained immediately: where I is the identity matrix of suitable dimension.
The error in the reconstructed field is seen to depend on both the approximation basis Φ and the measurement operator H. The measurement operator is entirely defined by the sensor location, and it does not depend on the basis considered to approximate the field. It is thus clear that, to reduce (the expectation of) the reconstruction error, the approximation basis must be informed both by the dataset {x i } i and the sensors available, through H. For example, poorly located sensors will lead to a large set of x i to lie in the nullspace of H, preventing their estimation, while the coefficients of certain approximation modes may be affected by the observation H x i of certain realizations x i being severely amplified by (H Φ) + if the approximation basis is not carefully chosen.
This remark can be interpreted in terms of the control theory concept of observability of the basis modes by the sensors. Most papers in the literature focus their attention on deriving an approximation basis leading to a good representation [13,71,50], i.e., such that the training set is well approximated in the k-dimensional basis {φ j } j , x ≈ Φ ν. But how well the associated coefficients ν = ν (s) are informed by the measurements is usually overlooked when deriving the basis. In practice, the decoupling between learning an approximation basis and learning a map to the associated coefficients often leads to a performance bottleneck in the estimation procedure. Enforcing observability of the approximation basis by the sensors is key to a good recovery performance and can dramatically improve upon unsupervised methods, as shown in [51].

SHALLOW LEARNING FOR FLOW RECONSTRUCTION
Shallow learning techniques are widely used for flow reconstruction. For instance, the approximation based approach for flow reconstruction, outlined in Section 2, can be considered to have two levels. The first level is concerned with computing an approximation basis, while the second level performs a linear weighted combination of the basis elements to estimate the high-dimensional flow field. Such shallow learning techniques are easy to train and tune. In addition, the levels are often physically meaningful, and they may provide some interesting insights into the underlying mechanics of the system under consideration. However, the recent success of deep learning has put shallow learning somewhat out of focus [8,44,65]. Indeed, the expressive power of deep architectures has pushed forward many tasks in computer vision and language processing. The high expressive power is due to a deep architecture design which has a large number of hidden layers. Particularly, in computer vision, the success is centered around convolution layers, which exhibit sparse connectivity of the neurons. These layers are augmented with non-linear activation functions and additional pooling layers. Several theoretical results support some of the advantage of deeper architectures [20,10,54,53]. Thus, the reader may wonder why we advocate shallow architectures for flow reconstruction? Among other things, for the applications in which we are interested, deep learning has the following downsides: • Computation: Deep architectures require tremendous amounts of computational power.
In the era of high-end graphic processing units (GPU), this is somewhat less of an issue, yet training can still be costly if the input data are high-dimensional.
• Tuning: Regularization in its various forms can be used to ease the issue of overfitting by limiting the complexity of the network. However, in practice, this requires "fiddling" with a large number of knobs (i.e., hyper-parameters) [7]. Generally, deeper architectures have been shown to be more sensitive to the choice of hyper-parameters. This remains a challenge even in light of recent progress in understanding generalization and overfitting in deep networks [59,5].
• Data: The more critical issue is that deep architectures are greedy, i.e., a large number of training examples is required in order to learn a function which generalizes to new data points. This is because deep nets are typically over-parametrized and tend to interpolate the data too closely [7,48,16]. Hence, the more examples used for training, the better the generalization error [59]. Further, there is a critical difference between standard machine learning benchmark datasets and scientific datasets (such as those we consider here). The former datasets, including MNIST, CIFAR10 and ImageNet, provide a large number of low-resolution training examples. In sharp contrast, scientific applications often generate a small amount of high-dimensional (high-volume) data, yet we face the situation that labeled examples are in short supply. For instance, recordings for climate data date back only so many years. Fortunately, scientific data often feature more structure, which can render an easier learning task. Hence, the hope is that shallow learning techniques perform better in the scientific data setting. Indeed, several results show that shallow learning is better suited for limited data than deeper networks [64,21,47,39]. Interestingly, SNNs have also successfully been used for applications arising in area of fluid mechanics, both before the recent hype over deep learning [30,55] and also more recently [33,6].
In the following, we show that the performance for flow reconstruction problems can be greatly improved by adding just one additional layer of complexity. This means that (instead of using a very shallow learning approach, as in traditional scientific methods) we explore architectures with one additional stage.

A SHALLOW DECODER FOR FLOW RECONSTRUCTION
We can define a neural network (NN) with K layers as a nested set of functions where R(·) : R → R denotes a coordinate-wise scalar (non-linear) activation function and W denotes a set of W k k weight matrices, k = 1, ..., K, with matching dimensions. NN-based learning provides a flexible framework for estimating the relationship between quantities from a collection of samples. Here, we consider SNNs, which are considered to be networks with very few, often only one, or even no, hidden layers, i.e., K is very small.
In the following, an estimate of a vector y is denoted as y, while y denotes dummy vectors upon which one optimizes. Relying on a training set with n examples x i and corresponding sensor measurements s i , we aim to learn a function F : s → x belonging to a class of neural networks F which minimizes the misfit in an Euclidean sense, over all sensor measurements We assume that only a small number of training examples is available. Further, no prior information is assumed to be available, and the estimation method is purely data-driven. Importantly, we assume no knowledge about the underlying measurement operator which is used to collect the sensor measurements. Further, unlike most classical methods for flow reconstruction, such as those discussed in Sec. 2, this NN-based learning methodology allows the joint learning of both the modes and the coefficients.

Architecture
We now discuss some general principles guiding the design of a good network architecture for flow reconstruction. These considerations lead to the following nested nonlinear function The architecture design is guided by the paradigm of simplicity. Indeed, the architecture should enable fast training, little tuning, and offer an intuitive interpretation.
Recall that the interpretability of the flow field estimate is favored by representing it in a basis of moderate size, whose modes can be identified with spatial structures of the field. This means, the estimate can be represented as a linear combination of k modes {φ j } j , weighted by coefficients {ν j } j , see Eq. (1). These modes are a function of the inputs. This naturally leads to consider a network in which the output x is given by a linear, fully connected, last layer of k inputs, interpreted as ν. These coefficients are informed by the sensor measurements s in a nonlinear way.
The nonlinear map s → ν can be described by a hidden layer, whose outputs ψ are hereafter termed measurement features, in analogy with kernel-based methods, where raw measurements s are nonlinearly lifted as extended measurements to a higher-dimensional space. In this architecture, the measurement features ψ essentially describe nonlinear combinations of the input measurement s. The nonlinear combinations are then mapped to the coefficients ν associated with the modes φ. While the size of the output layer is that of the discrete field x, the size of the last hidden layer (ν) is chosen and defines the size k of the dictionary Φ. This size can be estimated from the data {x i } i by dimensionality estimation techniques [36,29]. Restricting the description of the training data to a low-dimensional space is of potential interest to practitioners who may interpret the elements of the resulting basis in a physically meaningful way. The additional structure allows one to express the field of interest in terms of modes that practitioners may interpret, i.e., relate to some physics phenomena such as traveling waves, instability patterns (e.g., Kelvin-Helmholtz), etc.
In contrast, the size of the first hidden layer describing ψ is essentially driven by the size of the input layer (s) and the number of nonlinear combinations used to nonlinearly inform the coefficients ν. The general shape of the network then bears flexibility in the hidden layers. A popular architecture for decoders consists of non decreasing layer sizes, so as to increase continuously the size of the representation from the low-dimensional observations to the high-dimensional field. We can model F as a shallow neural network with two hidden layers ψ and ν, followed by a linear output layer Ω.
Two types of hidden layers, namely fully-connected (FC) and convolution layers can be considered. The power of convolution layers is key to the success of recent deep learning architectures in computer vision. However, in our problem, we favor fully-connected layers. The reason is two-fold: (i) our sensor measurements have no spatial ordering; (ii) depending on the number of filters, convolution layers require a large number of examples for training, while we assume that only a small number of examples are available for training. Thus, the first and second hidden layers take the form , where W denotes a dense weight matrix and b is a bias term. The function R(·) denotes an activation function used to introduce nonlinearity into the model as discussed below. The final linear output layer simply takes the form of x = Ω(z ν ) := Φz ν + b Φ , where we interpret the columns of the weight matrix Φ as modes. In summary, the architecture of our shallow decoder can be outlined as Depending on the dataset, we need to adjust the size of each layer. Here, we use narrow rather than wide layers. Prescribing the size of the output layer restricts the dimension of the space in which the estimation lies, and it effectively regularizes the problem, e.g., filtering-out most of the noise which is not living in a low-dimensional space. Beyond robustness with respect to noise, reducing the dimension brings several additional benefits, including faster learning and fewer suboptimal local minima.
The rectified linear unit (ReLU) activation function is among the most popular choices in computer vision applications, owing to its favorable properties [35]. The ReLU activation, illustrated in Figure 3a, is defined as the positive part of a signal z: The transformed input signal is also called activation. While the ReLU activation function performs best on average in our experiments, there are other choices. For instance, we have considered the Swish [2] and SoftShrinkage activation function, also illustrated in Figure 3. These two activation functions can be fine-tuned via an additional hyper-parameter and there are potential situations in which these activation functions outperform ReLU. Interestingly, different activation functions considerably affect the modes (i.e., columns of the weight matrix Φ), as shown in Figure 4.

Regularization
Overfitting is a common problem in machine learning and occurs if a function interpolates a limited set of data points too closely. In particular, this is a problem for deep neural networks which often have more neurons (trainable parameters) than can be justified by the limited amount of training examples which are available. There is increasing interest in characterizing and understanding generalization and overfitting in NNs [59,5]. Hence, additional constraints are required to learn a function which generalizes to new observations that have not been used for training. Standard strategies to avoid overfitting include early stopping rules, and weight penalties (L 2 regularization) to regularize the complexity of the function (network). In addition to these two strategies, we use also batch normalization (BN) [40] and dropout layers (DL) [66] to improve the convergence and robustness of the shallow decoder. This yields the following architecture: Regularization, in its various forms, requires one to "fiddle" with a large number of knobs (i.e., hyper-parameters). However, we have found that SNNs are less sensitive to the particular choice of parameters; hence, SNNs are easier to tune. Batch normalization. BN is a technique to normalize (mean zero and unit standard deviation) the activation. From a statistical perspective, BN eases the effect of internal covariate shifts [40]. In other words, BN accounts for the change of distribution of the output signals (activation) across different mini batches during training. Each BN layer has two parameters which are learned during the training stage. This simple, yet effective, prepossessing step allows one to use higher learning rates for training the network. In addition it also reduces overfitting owing to its regularization effect. Dropout layer. DL helps to improve the robustness of a NN. The idea is to switch off (drop) a small fraction of randomly chosen hidden units (neurons) during the training stage. This strategy can be seen as some form of regularization which also helps to reduce interdependent learning between the units of a fully connected layer. In our experiments the drop ratio is set to p = 10%.

Optimization
Given a training set with n targets {x i } i and corresponding sensor measurements {s i } i , we minimize the misfit between the reconstructed quantity x = F(s) and the observed quantity x, in terms of the squared L 2 -norm The second term on the right hand side introduces L 2 regularization to the weight matrices, which is controlled via the parameter λ. It is well-known that L 2 -norm is sensitive to outliers; and the L 1 -norm can be used as a more robust loss function. Alternatively, a popular option is the Huber norm (smooth L 1 -loss), leading to the following optimization problem The tuning parameter κ controls the threshold. The Huber loss functions grow at a linear rate for residuals outside the thresholding parameter κ, rather than quadratically. This can reduce the influence of large deviations when learning the decoder. Further, it has been reported that this loss function prevents exploding gradients in some cases [34]. Thus, the Huber loss may be an interesting alternative. We use the ADAM optimization algorithm [43] to train the shallow decoder, with learning rate 10 −2 and weight decay 10 −4 (also known as L 2 regularization). The learning rate, also known as step size, controls how much we adjust the weights in each epoch. The weight decay parameter is important since it allows one to regularize the complexity of the network. In practice, we can improve the performance by changing the learning rate during training. We decay the learning rate by a factor of 0.9 after 100 epochs. Indeed, the reconstruction performance in our experiments is considerably improved by this dynamic scheme, compared to a fixed parameter setting. In addition, we decrease the weight decay by a factor of 0.8. Further, we use a relatively large batch size, since we have only a limited amount of data available for training. Overall, in our experiments, ADAM shows a better performance than stochastic gradient decent (SGD) with momentum [67] and averaged SGD [60]. The hyper-parameters can be fine tuned in practice, but our choice of parameters works reasonably well for several different examples. Note that we use the method described by [38] in order to initialize the weights. This initialization scheme is favorable, in particular because the output layer is high-dimensional.

EMPIRICAL EVALUATION
We evaluate our methods on three classes of data. First, we consider a periodic flow behind a circular cylinder, as a canonical example of fluid flow. Then, we consider the weekly mean sea surface temperature (SST), as a second and more challenging example. Finally, the third and most challenging example we consider is a forced isotropic turbulence flow.
As discussed in Section 1, the shallow decoder requires that the training data represent the system, in the sense that they should comprise samples drawn from the same statistical distribution as the testing data. Indeed, this limitation is standard to data-driven methods, both for flow reconstruction and also more generally. Hence, we are mainly concerned with exploring reconstruction performance and generalizability for interpolation tasks rather than for extrapolation tasks. In our third example, however, we demonstrate the limitations of the the shallow decoder, illustrating difficulties that arise when one tries to extrapolate, rather than interpolate, the flow field. Figure 5 illustrates the difference between the two types of tasks.
In the first two example classes of data, the sensor information is a subset of the high-dimensional flow field, i.e., the measurement operator H ∈ R p×m only has one non-zero entry in rows corresponding to the index of a sensor location. Letting J ∈ [1, m] p ⊂ N p be the set of indices indexing the spatial location of the sensors, the measurement operator is such that that is, the observations are simply point-wise measurements of the field of interest. In the above equation, x J is the restriction of x to its rows indexed by J . In this paper, no attempt is made to optimize the location of the sensors. In practical situations, they are often given or constrained by other considerations (wiring, intrusivity, manufacturing, etc.). We use simply uniform random locations in our examples. The third example class of data demonstrates the SD using sub-gridscale measurements.  The quality of the reconstruction accuracy is quantified in terms of the normalized root-meansquare residual error NME = x − x 2 x 2 , denoted in the following as "NME." However, this measure can be misleading if the empirical mean is dominating. Hence, we consider also a more sensitive measure which quantifies the reconstruction accuracy of the deviations around the empirical mean. We define this measure as where x and x are the fluctuating parts around the empirical mean. In our experiments, we average the errors over 30 runs for different sensor distributions.

Setup for our empirical evaluation
Here, we provide details about the concrete network architectures of the shallow decoder, which are used for the different examples. The networks are implemented in Python using PyTorch; and research code for flow behind the cylinder is available via https://github.com/erichson/ShallowDecoder. Tables 1-3 show the details. For each example we use a similar architecture design. The difference is that we use a slightly wider design (more neurons per layer) for the SST dataset and the isotropic flow. That is because we are using a larger number of sensors for these two problems, and thus we need to increase the capacity of the network. In each situation, the learning rate is set to 10 −2 with a decay rate of 0.9. The weight decay is set to 10 −4 with a decay rate of 0.8. We stop training after 4000 epochs.    Table 3: Architecture of the SD for isotropic flow. Here, the batch size is set to 200.

Fluid flow behind cylinder
The first example we consider is the fluid flow behind a circular cylinder, at Reynolds number 100, based on cylinder diameter, a canonical example in fluid dynamics [57]. The flow is characterized by a periodically shedding wake structure and exhibits smooth, large scale, patterns. A direct numerical simulation of the two-dimensional Navier-Stokes equations is achieved via the immersed boundary projection method [68,19]. In particular, we use the fast multidomain method [19], which simulates the flow on five nested grids of increasing size, with each grid consisting of 199 × 449 grid points, covering a domain of 4 × 9 cylinder diameters on the finest domain. We collect 151 snapshots in time, sampled uniformly in time and covering several periods of vortex shedding. For the following experiment, we use cropped snapshots of dimension 199 × 384 on the finest domain, as we omit the spatial domain upstream to the cylinder. Further, we split the dataset into a training and test set so that the training set comprises the first 100 snapshots, while the remaining 51 snapshots are used for validation. Note that different splittings (interpolation and extrapolation) yield nearly the same results since the flow is periodic.

Varying numbers of random structured point-wise sensor measurements
We investigate the performance of the shallow decoder using varying numbers of sensors. A realistic setting is considered in that the sensors can only be located on a solid surface. The retained configuration aims at reconstructing the entire vorticity flow field from information at the cylinder surface only. The results are averaged over different sensor distributions on the cylinder downstream-facing surface and are summarized in Table 4. Further, to contextualize the precision of the algorithms, we also state the standard deviation in parentheses.
The shallow decoder shows an excellent flow reconstruction performance compared to traditional methods. Indeed, the results show that very few sensors are already sufficient to get an accurate approximation. Further, we can see that the shallow decoder is insensitive to the sensor location, i.e., the variability of the performance is low when different sensor distributions on the cylinder surface are used. In stark contrast, this simple setup poses a challenge for the pod method, which is seen to be highly sensitive to the sensor configuration. This is expected since poorly located sensors lead to a large probability that the vorticity field x i lies in the nullspace of H, preventing its

Sensors
Training Set Test Set NME NFE NME NFE  estimation, as discussed in Section 2. While regularization can improve the robustness slightly, the pod plus approach still requires about at least 15 sensors to provide accurate estimations for the high-dimensional state-space of the flow where the shallow decoder exhibits a good performance with as few as 5 sensors. Note that the traditional methods could benefit from optimal sensor placement [50]; however, this is beyond the scope of this paper.    Figure 6 provides visual results for two specific sensor configuration using 5 sensors. The second configuration is challenging for pod, which fails to provide an accurate reconstruction. pod plus provides a more accurate reconstruction of the flow field. The shallow decoder outperforms the traditional methods in both situations. Further insights can be gained by examining the singular value spectrum of the original and reconstructed data constituted of a matrix collecting snapshots at different time instants; see Figure 7. The spectrum of the reconstructed flow data using the shallow decoder is seen to closely approximate the true spectrum, while the spectrum of the data reconstructed using pod show provides a very poor approximation.

Non-linear sensor measurements
So far, the sensor information consisted of pointwise measurements of the local flow field so that the j-th measurement is given by s (j) = H j x = δ τ j [x] = x (j) , j = 1, . . . , p, with δ τ j a Dirac distribution centered at the location of the j-th sensor and s (j) and x (j) the j-th component of s and x respectively. We now consider nonlinear measurements to demonstrate the flexibility of the shallow decoder. Here, we consider the simple setting of squared sensor measurements: denotes the Hadamard product. Table 5 provides a summary of the results, using 10 sensors. The shallow decoder is agnostic to the functional form of the sensor measurements, and it achieves nearly the same performance as in the linear case above. The average reconstruction accuracy for the test set increases only by about 1%. The POD-based methods fail for this task since they are linear techniques.

Sensors
Training Set Test Set NME NFE NME NFE  Table 5: Performance for estimating the flow behind a cylinder using nonlinear sensor measurements. The standard POD-based method fails for this task. pod plus is able to reconstruct the flow filed, yet the estimation quality is poor. In contrast, the SD method performs well.

Noisy sensor measurements
To investigate further the robustness and flexibility of the shallow decoder, we consider flow reconstruction in the presence of additive white noise. While this is not of concern when dealing with flow simulations, it is a realistic setting when dealing with flows obtained in experimental studies. Table 6 lists the results for both a high and low noise situation with linear measurements. By inspection, the performance of the shallow decoder outperforms classical techniques. In the high noise case, with a signal-to-noise ratio (SNR) of 10, the average relative reconstruction error for   Table 6: Performance for estimating the flow behind a cylinder in presence of white noise, using 10 sensors. pod fails for this task, while pod plus shows a better performance. The SD shows to be robust to noisy sensor measurements and outperforms the traditional techniques.  the test set is about 27% for the shallow decoder. For a SNR of 50, the relative error is as low as 17%. Note that we here use an additional dropout layer (placed after the first fully-connected layer) to improve the robustness of the shallow decoder. In contrast, standard pod fails in both situations. Again, the pod plus method shows improved results over the standard pod. However, the visual results in Figure 8 show that the reconstruction quality of the shallow decoder is favorable. The shallow decoder shows a clear advantage and a denoising effect. Indeed the reconstructed snapshots allow for a meaningful interpretation of the underlying structure. The shallow decoder can thus be seen as a valuable tool for the reconstruction of fluid flows in the presence of noise. Figure 9 summarizes the performance of the shallow decoder for varying measurement configurations (number of sensors, linear or nonlinear, noise). The advantage of the shallow decoder compared to the traditional POD based techniques is pronounced. It can be seen that the performance of the traditional techniques is patchy, i.e., the reconstruction quality is highly sensitive to the sensor location. While regularization can mitigate a poor sensor placement design, a relatively larger number (> 15) of sensors is required in order to achieve an accurate reconstruction performance. More challenging situations such as nonlinear measurements and sensor noise pose a challenge for the traditional techniques, while the shallow decoder shows to be able to reconstruct dominant flow features in such situations. The computational demands required to train the shallow decoder are minimal, compared to training deep architectures, i.e., the time for training remains below two minutes for this example, using a modern GPU.

Sea surface temperature using random point-wise measurements
The second example we consider is the more challenging sea surface temperature (SST) dataset. Complex ocean dynamics lead to rich flow phenomena, featuring interesting seasonal fluctuations. While the mean SST flow field is characterized by a periodic structure, the flow is non-stationary. The dataset consists of the weekly sea surface temperatures for the last 26 years, publicly available from the National Oceanic & Atmospheric Administration (NOAA). 1 The data comprise 1483 snapshots    We consider the performance of the shallow decoder using varying numbers of random sensors scattered across the spatial domain. The results are summarized in Table 7. We observe a large discrepancy between the NME and NFE error. This is because the long-term annual mean field accounts for the majority of the spatial structure of the field. Hence, the NME error is uninformative with respect to the performance of reconstruction methods. In terms of the NFE error the POD based reconstruction techniques is shown to fail to reconstruct the high-dimensional flow field using limited sensor measurements. In contrast, the shallow decoder demonstrates an excellent reconstruction performance both using 32 and 64 measurements. Figure 10 shows visual results to support these quantitative findings.

Turbulent flow using sub-gridscale measurements
The final example we consider is the velocity field of a turbulent isotropic flow. If the sensor measurements s are acquired on a coarse but regular grid, then the reconstruction task may be considered as a super-resolution problem [72,31,14]. There are a number of direct applications of super-resolution in fluid mechanics centered around sub-gridscale modeling. Because many fluid flows are inherently multiscale, it may be prohibitively expensive to collect data that captures all spatial scales, especially for iterative optimization and real-time control [12]. Inferring small-scale flow structures below the spatial resolution available is an important task in large eddy simulation (LES), climate modeling, and particle image velocimetry (PIV), to name a few applications. Deep learning has recently been employed for super-resolution in fluid mechanics applications with promising results [32].
Here, we consider data from a forced isotropic turbulence flow generated with a direct numerical simulation using 1, 024 3 points in a triply periodic [0, 2π] 3 domain. For the following experiments, we are using 800 snapshots for training and 200 snapshots for validation. The data spread accross about one large-eddy turnover time. The full dataset is provided as part of the Johns Hopkins Turbulence Database [45,58].

Interpolation
Unlike the examples considered in Section 5, the isotropic turbulent flow is non-periodic in time and highly non-stationary. Thus, this dataset poses a challenging task, even for interpolation. Figure 11 shows visual example for this problem. Note that in our setting the mean grid values are used as inputs, while in the classical super-resolution problem the low-resolution image, shown in Figure 11b, is used as an input. The quality of the estimated high-dimensional flow field is       excellent, despite the challenging problem. Table 8 quantifies the performance for varying numbers of sub-gridscale measurements.

Extrapolation
Next, we illustrate the limitation of the shallow decoder. Indeed, it is important to stress that the shallow decoder cannot be used for extrapolating highly non-stationary fluid flows. To illustrate this issue, Figure 12 shows three flow fields at different temporal locations. First, Figure 12a shows a test example, which is close in time to the training set. In this case, the shallow decoder is able to extrapolate the flow field with high accuracy. The reconstruction quality drops for snapshots which are further away in time, as shown in Figure 12c. Finally, Figure 12e shows that extrapolation fails if the test example is far away from the training set in time, i.e., the flow field is not drawn from the same statistical distribution as the training examples are.

DISCUSSION
The emergence of sensor networks for global monitoring (e.g., ocean and atmospheric monitoring) requires new mathematical techniques that are capable of maximally exploiting sensors for state estimation and forecasting. Emerging algorithms from the machine learning community can be integrated with many traditional scientific computing approaches to enhance sensor network capabilities. For many global monitoring applications, the placement of sensors can be prohibitively expensive, thus requiring mathematical techniques such as the one proposed here, which can exploit a hyper reduction in the number of sensors while maintaining required performance characteristics.
This work demonstrates the enhanced robustness and accuracy of fluid flow field reconstruction by using a shallow learning based methodology. We have explored this approach on a range of example flow fields of increasing complexity. The mathematical formulation presented is significantly different from what is commonly used in flow reconstruction problems, e.g., gappy interpolation with dominant POD modes.
We proposed a shallow decoder with two hidden layer for the problem of flow reconstruction in order to achieve an improved reconstruction performance. During our study, we also compared the shallow decoder to deep architectures, which yield no significant improvement (results are omitted). More concretely, we considered deep convolution networks (DCN) with three to five hidden deconvolutional layers, as well as residual network (resnet) architectures. The reconstruction performance was marginally better for the simple flow behind the cylinder. However, we observed that the shallow decoder shows a favorable performance in all other situations. A further advantage we observed in our experiments is that shallow architectures are more robust to sensor noise. Moreover, the features extracted for reconstruction are highly interpretable, potentially allowing for enhanced scientific understanding of the system measured. Table 9 shows a qualitative comparison of our initial experiments for flow reconstruction. Our results show that shallow architectures are more favorable for limited sensor and limited data settings. In conclusion, we advocate a regression towards more shallow networks for flow reconstruction and more generally for scientific applications with limited data.
Future work aims to leverage the underlying laws of physics in flow problems to further improve the efficiency. In the context of flow reconstruction or, more generally, observation of a high-dimensional physical system, insights from the physics at play can be exploited [62,61]. In particular, the dynamics of many systems do indeed remain low-dimensional and the trajectory of their state vector lies close to manifold whose dimension is significantly lower than the ambient dimension. Moreover, the features exploited from the shallow decoder network can also be integrated in reduced order models (ROMs) for forecasting predictions [9]. In many high-dimensional systems where ROMs are used, the ability to generate low-fidelity models that can be rapidly simulated has revolutionized our ability to model such complex systems, especially in application of complex flow fields. The ability to rapidly generate alternative low-rank feature spaces to POD generates new possibilities for ROMs using limited sampling and limited data. This aspect of the shallow decoder will be explored further in future work.