{'fraction_non_alphanumeric': 0.025382858109823166, 'fraction_numerical': 0.012014552838649632, 'mean_word_length': 4.636623748211731, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Linearized Learning with Multiscale Deep Neural Network for Stationary Navier-Stokes Equations with Oscillatory Solutions

In this paper, we combine a linearized iterative method with multi-scale deep neural network to compute oscillatory flows for stationary Navior-Stokes equation in complex domains. The multiscale neural network converts the high frequency components in target to low frequency components before training, thus accelerating the convergence of training of neural network for all frequencies. To solve the stationary nonlinear Navier-Stokes equation, we introduce the idea of linearization of Navier-Stokes equation and iterative methods to treat the nonlinear convection term. Three forms of linearizations will be considered. First we will conduct a benchmark problem of the linearized schemes in comparison with schemes based directly on the nonlinear PDE. Then, a Navier Stokes flow with high frequency components in a 2-D domain with a hole are learned by the linearized multiscale deep multiscale neural network. The results show that multiscale deep neural network combining with the linearized schemes can be trained fast and accurately.


Introduction
Deep neural network (DNN) machine learning has been studied as alternative numerical methods for solving partial differential equations arising from fluid dynamics [1] and wave propagations [2] . DNN has the potential of a flexible meshless method to solve governing equations from fluid and solid mechanics applications as an alternative method to traditional finite element method. Moreover, it has shown much power in handling high dimensional parabolic PDEs [3,4].Recent work on DNN have shown a particular frequency dependence performance in learning solution of PDEs and fitting of functions. Namely, the lower frequency components of the solution are learned first and quickly than the higher frequency components [5]. Several attempts have been made to remove such a frequency bias for the DNN performances. The idea is to convert the higher frequency content of the solution to a lower frequency range so the conventional DNN can learn the solution. One way to achieve this goal is to use a phase shift while the other way is to introduce a multiscale structure of the DNN where neural network with different scale will target different range of the frequency in the solutions. The PhaseDNN has been shown to be very effect for high frequency wave propagations while the MscaleDNN has been used to learn highly oscillatory Stokes flow solutions in complex domains.
In this paper, we will study how to extend the MscaleDNN for the nonlinear Navier Stokes equations. A direct application of MscaleDNN to the nonlinear PDEs does not produce the dramatic improvement for the linear Stoke equations over the conventional DNN. The learning of the solution of linear PDEs via least square residual of the PDEs is in some sense equivalent to the fitting problems in the frequency domain from the Parsevel equality of the Fourier transform. So it is understandable that a multiscale DNN's performance for a fitting problem also holds for the solution of linear PDEs. Taking this cue for the MscaleDNN's improvement for the linear Stokes problem, we will develope a linearized learning procedure for the Navier-Stokes equations by integrating a linearization of the Navier-Stokes equation in the loss function of the MscaleDNN and dynamically updating the linearization as the learning is being carried out. Numerical results have shown the convergence of this approach and it produces highly accurate approximation to oscillatory solutions of the Navier-Stokes equations.
The rest of the paper will be organized as follows. Section 2 will introduce the multiscale DNN structure and then three linearized learning for the Navier-Stokes equation will be proposed in Section 3. Numerical tests of the linearized learning schemes will be conducted for 2-D oscillatory flows in a domain containing a cylinder in Section 4. Finally, conclusion and future work will be discussed in Section 5.

Multiscale neural network algorithm
In order to improve the capability of the DNN to represent functions with multiple scales, we will apply the MscaleDNN [6], which consists of a series of parallel normal sub-neural networks. Each of the sub-networks will receive a scaled version of the input and their outputs will then be combined to make the final out-put of the MscaleDNN (refer to Fig. 1). The individual subnetwork in the MscaleDNN with a scaled input is designed to approximate a segment of frequency content of the targeted function and the effect of the scaling is to convert a specific high frequency content to a lower frequency range so the learning can be accomplished much quickly. Due to the radial scaling used in the MscaleDNN, it is specially fitting for approximation of high dimensional parabolic equations.
where W [1] to W [L−1] and b [1] to b [L−1] are the weight matrices and bias unknowns, respectively, to be optimized via the training, σ(x) is the activation function. In this work, the following plane wave activation function will be used for its localized frequency property [1], For the input scales, we could select the scale for the i-th sub-network to be i (as shown in Fig. 1) or 2 i−1 . Mathematically, a MscaleDNN solution f (x) is represented by the following sum of sub-networks f θ n i with network parameters denoted by θ n i (i.e., weight matrices and bias) where α i is the chosen scale for the i-th sub-network in Fig. 1. For more details on the design and discussion of the MscaleDNN, please refer to [6].
3 Stationary Navier Stokes Equations and Linearization of the nonlinear term

Stationary Navier Stokes Equations
The following two dimensional stationary Navier-Stokes equation will be solved by the deep neural networks, where Ω is an open bounded domain in R 2 . The MscaleDnn solution will be found as in the traditional least square finite element method where the solution is obtained by minimizing a loss function in terms of the residual of Navier Stokes equation.
To solve the NS stokes in a least square approach, as usual, the PDEs (4) is written in a system of first order equations in [1]. A velocity-gradient of velocity-pressure formulation will be used for the Navier Stokes equations.
To obtain the governing equation for the pressure, we take divergence on both sides of the Navier Stokes equation to arrive at the following Poisson equation, Then, a loss function can be defined as However, it will be shown that the training of the network based on the formulation 5 including the Nonlinear first order system and the Poisson equation 6 converges slowly if at all, as shown by the numerical results in in section 4.1.

Three Linearized Learning Schemes
To speed up the convergence of the training, we propose to introduce an iterative solution procedure for the nonlinear NS equation so each step of the training is done for a linearized version of the Navier-Stokes equation and the linearization [7] will be updated as the training progresses.
Assume the current learned solutions are (u n , v n ), the next functions to be learnt are denoted as (u n+1 , v n+1 ), then we can linearize the Navier-Stokes equation in the following three schemes.
• Scheme 1: Gradients of velocity are fixed.
In this scheme, we use the currently learned solutions with a few epochs training to replace the gradients in the nonlinear term.
In this scheme, we fix the velocity term in nonlinear term to be the current learned velocity.
A average of the previous two schemes is taken to replace the nonlinear convection term in the NS equation for this scheme.
Loss function for Scheme 1: Loss ∇u = Res u + Res v + Bdry u + Res p , Loss function for Scheme 2: Loss u = Res u + Res v + Bdry u + Res p , Loss function for Scheme 3: Loss mix = Res u + Res v + Bdry u + Res p ,

Linearized Learning Algorithms with MscaleDNN
Our linearized MscaleDNN learning algorithms for the NS equation are designed with the following steps.
• Initialize u 0 ,u temp and p • i = 1 • threshold = 1e12 • While i < Epochs fix u temp ,using loss one of the loss function in (11) or (12) or (13) to train u i and p -Every 5 epoch check the loss. If the loss is less than the threshold * u temp = u i * threshold = Loss 4 Numerical Results

A benchmark problem
We first consider the problem in a rectangle domain Ω = [0, 2] × [0, 1] with one cylinder hole centered at 0.7, 0.5 whose radius is 0.2, with the analytical solution of the incompressible Navier-Stokes equations given as follows: We will compare the decay of the loss functions with a fully connected NN based on the nonlinear NS equation (7) and the loss of the 3 linearized schemes (11) or (12) or (13).
As a benchmark problem, we consider the frequency n = 1, m = 2. The source term f is obtained by substituting the exact solution into the Navier Stokes equation 4. We set the viscosity ν = 0.05 and compare the performance of algorithms using the VgVP formulation with the loss function in (7) and three linearized learning schemes. We randomly sample 160000 points inside Ω and 16000 points inside ∂Ω during each epoch. In the learning process, we set batch size to be 3200 inside domain and 320 on boundary for each step. The deep neural network solutions obtained by minimizing different loss functions in (11) − (13) are compared in Figure 2. The results show that all three linearized learning neural networks converge in 300 epochs for all schemes while learning using the loss function (7) for the nonlinear Navier-Stokes equations fails to (top line in Figure 2 ). All the schemes use 4 hidden layer, 100 hidden neurons, fully connected neural networks for u, p.

Oscillating flows
In our previous work, it has been shown that a multiscale DNN will be needed to learn oscillatory solutions for the Stokes equations. So the linearized learning schemes with mscale deep neural network will be also used for Navier-Stokes equations with oscillatory solutions. Here we use the same domain like the benchmark problem but the frequencies now are taken to be   (7) for the nonlinear NS does not coverge within the same number of epoches.
We consider Scheme 2 of the linearized learning algorithms where the previous velocity are used to linearized the convection velocity. Figure 5 and 8 show the results of network after 1000 epoch training. The multiscale deep neural networks has 8 scales: {x, 2x, 4x, 8x, 32x, 64x, 128x}, whose subnetworks contain 4 hidden layers and 128 hidden neurons in each layer. As comparison, we use a 4-layer fully connected neural network with 1024 hidden neurons combining scheme 2 for this oscillating flows. The results are shown in Fig 6. Fig 7 shows the errors of these 2 different neural network structures along line y = 0.7. The MscaleDNN improves the accuracy of the pressure field compared with the fully connected neural network while both DNN performs similarly for the velocity fields. We expect that the MscaleDNN will perform substantially better for higher frequency flow fields.

Conclusion and future work
In this paper we proposed three linearized learning schemes to solve the stationary highly oscillatory Navier-Stokes flows with multiscale deep neural networks and showed the acceleration of convergence of the schemes are substantial, which demonstrate the capability of the Mscale deep neural net-  On the other hand, there are much more works to be done for these linearized learning methods, among them the most important one is to understand the convergence property of these schemes. The applications of these schemes to other nonlinear PDEs should be considered. Another challenging future work is to consider the time dependent Navier-Stokes equation, which will be explored in the near future.