{'fraction_non_alphanumeric': 0.03299185915163791, 'fraction_numerical': 0.014957348186811046, 'mean_word_length': 4.8007230004518755, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
VI-PINNs: Variance-involved Physics-informed Neural Networks for Fast and Accurate Prediction of Partial Differential Equations

Although physics-informed neural networks(PINNs) have progressed a lot in many real applications recently, there remains problems to be further studied, such as achieving more accurate results, taking less training time, and quantifying the uncertainty of the predicted results. Recent advances in PINNs have indeed significantly improved the performance of PINNs in many aspects, but few have considered the effect of variance in the training process. In this work, we take into consideration the effect of variance and propose our VI-PINNs to give better predictions. We output two values in the final layer of the network to represent the predicted mean and variance respectively, and the latter is used to represent the uncertainty of the output. A modified negative log-likelihood loss and an auxiliary task are introduced for fast and accurate training. We perform several experiments on a wide range of different problems to highlight the advantages of our approach. The results convey that our method not only gives more accurate predictions but also converges faster.


Introduction
Physics-informed neural networks(PINNs) [1] have been proved to be efficient and powerful to handle problems from different areas with the development of deep neural networks(DNNs) [2][3][4]. By combining underlying physical information usually described by partial differential equations(PDEs) with neural networks, PINNs performed better than traditional data-driven neural networks while using fewer data.
Despite the great progress in PINNs, how to effectively improve the performance of PINNs is still an issue worth studying. Bischof and Kraus [5] summarized that current research may be concluded into four approaches: modifying the structure, divide-and-conquer/domain decomposition, parameter initialization, and loss balancing. The results of [6,7] vividly conveyed to us that modifying structure does make PINNs get more knowledge to speed up convergence. The key idea of domain decomposition is to solve subproblems defined on smaller subdomains rather than the whole domain [8,9], and [10,11] put into together the PINNs and domain decomposition to improve the performance of vanilla PINNs. Novel algorithms for adaptive loss balancing were proposed in [5,12,13].
How to quantify the uncertainty in PINNs is another issue that needs to be studied. Bayesian NNs are one of the most appropriate methods for estimating uncertainty, where the posterior is computed based on a prior distribution and training data. The work of Yang et al. [14] is the combination of PINNs and Bayesian NNs, but compared with non-Bayesian NNs, the cost of computation is expensive. Dropout [15] is another method used for uncertainty estimation. During the training of NNs, we drop some neurons at random, and it seems that we are training different neural networks, so the final result is the average of the effects of diverse networks, which can be thought of as an ensemble of different models to some extent.
In this work, we propose our simple framework named variance-involved physics-informed neural networks(VI-PINNs). We hold the hopes and ideas that (i) the estimation of uncertainty in PINNs is bound to be simple but effective. Inspired by [16][17][18], we output two values in the final layer, which is the main idea of heteroscedastic neural network(HNN), to represent the mean and variance, and this is an improvement in the structure of vanilla PINNs; (ii) the uncertainty should be incorporated into the training of PINNs. We modify the negative log-likelihood loss(NLL) used in HNN to make it more suitable for PINNs, and instead of using the modified loss alone, we introduce it to the PINN loss as an auxiliary task to assist the training of vanilla loss, and this is an improvement in the loss function of vanilla PINNs. The main contributions of this paper are summarized below: 1. We modify the structure of vanilla PINNs to model the uncertainty in an explicit way. 2. We modify the NLL loss function commonly used in HNN to make it suitable for the training of vanilla PINNs.
3. We incorporate the modified loss function as an auxiliary task to help the convergence of vanilla PINNs.
The rest of the paper is organized as follows: A brief introduction to the background of PINNs, together with some related work, is presented in Section 2. In Section 3, we propose our variance-involved physics-informed neural networks(VI-PINNs). Computational results for different PDE problems are provided in Section 4. Finally, Section 5 concludes the paper.

Background
This section briefly reviews physics-informed neural networks(PINNs) in tackling partial differential equations(PDEs). Then some recent developments in improving the training efficiency and accuracy of PINNs along with quantification of uncertainty are covered.

Physics-Informed Neural Networks(PINNs)
It has been a long time since artificial neural networks were applied to solve ordinary and partial differential equations [19][20][21]. Recently, with the rapid development of deep learning, [1] incorporated the differential form of the partial differential equation into the design of the loss function. Consider a partial differential equation with initial and boundary conditions taking the following general form: where u(t, x) represents the solution of PDE, N [·] and B[·] are linear/nonlinear differential operators, Ω is a subset of R D , and ∂Ω is the boundary of Ω. PINNs consider approximating u(t, x) by a deep neural network u θ (t, x), so that the value of N [u θ (t, x)] and B[u θ (t, x)] can be computed readily and precisely using automatic differentiation [22,23]. The residual form of Equation (1) is defined as follows: The weights θ are trained by minimizing a loss function: where L r , L b , and L 0 are loss terms that penalize the PDE (1), boundary condition (2), initial condition (3), respectively: where {t i r , x i r } Nr i=1 are collocation points randomly sampled from the domain are initial condition points, and N r , N b , and N 0 denote the number of collocation, boundary, initial points, respectively. The structure of vanilla PINNs is shown in Figure 1.

Related work
In the previous section, the most basic idea of vanilla PINNs was described. Although PINNs have attracted many attentions in recent years, handling high dimensional and complex PDE problems remains challenging. Further studies need to be done to achieve higher training efficiency, more accurate prediction, and faster convergence speed.

Neural network architecture
Many people [24][25][26][27][28] modified neural network architectures to achieve a better result. In [24][25][26], boundary conditions are enforced more precisely by constructing special neural network architectures. Demo et al. [27] aimed to divide the neural network into a combination of smaller neural networks, thus isolating the relations between the outputs of the model. Yang et al. [28] used multiple neurons in the output layer so that knowledge regarding uncertainty can be imposed on the distribution formed by outputs.

Domain decomposition
Krishnapriyan et al. [29] proved that failure modes caused by soft regularization in PINNs can make the loss harder to optimize. They proposed two approaches to tackle these failure modes, one is to use curriculum regularization, where a good initialization is found to warm start the training, the other is to view the problem as a sequence-to-sequence(seq2seq) learning task to predict the solution at next time step, rather than learning the entire space-time, and this idea is also mentioned in [30][31][32].

Parameter initialization
Xavier initialization [33] is widely used by researchers to select appropriate initial weights and biases, but few consider the influence of initialization. Liu et al. [34] proposed a meta-learning initialization method based on labeled data, which allows PINNs to converge faster and predict more accurately.

Loss balancing
Different parts of the loss function tend to descend at different speeds using gradient descent, which restricts weights and biases to converge to the global optimal solution. So most of the further studies on PINNs focus on balancing different parts of loss function rather than treating them equally. The standard approach is to introduce weights into Equation (5): Wight and Zhao [31] pointed out that it is significant to enforce neural networks to fit the initial conditions exactly, especially for equations that can only be solved in the forward time direction. Accordingly, the loss function was redefined as L(θ) = L r (θ) + L b (θ) + C 0 L 0 (θ), where C 0 is a big positive constant that is tunable. In [12], the authors, drawing motivation from Adam [35], proposed a learning rate annealing procedure, which was designed to scale the loss by utilizing the back-propagated gradient statistics automatically. Yuan et al. [36] adopted an adaptive strategy, where the weights of components are proportional to the values of residuals, that is the weight of the smallest residual component will be normalized to 1, and greater weights will be assigned to larger components. SoftAdapt [37] balances the loss terms using the rate of change of each loss term between consecutive steps. Bischof and Kraus [5] proposed a novel method taking into consideration the advantages of [12] and [37], where gradient statistics were avoided and drastic changes in the loss space were reduced.

Uncertainty quantification
Quantifying uncertainty in PINNs is still a challenging problem. Yang et al. [14] proposed a Bayesian physics-informed neural network(B-PINN) to provide predictions and quantify the aleatoric uncertainty arising from the noisy data. In [38], the random parameter in the equation was represented as a stochastic process, introducing parametric uncertainty, dropout [15] was also used to estimate the approximation uncertainty, the parametric uncertainty, and the approximation uncertainty were referred to as total uncertainty. Yang and Perdikaris [39] employed latent variable models to construct probabilistic representations so that uncertainty can be quantified and propagated. Literature mentioned above all modeled the uncertainty in an implicit or indirect way, motivated by [16][17][18], we output two values in the final layer of vanilla PINNs, representing the predicted mean and variance directly, and we also incorporate the variance into the training of PINNs, which speeds up the convergence speed.

Methodology
The goal of this paper is to take into consideration the influence of uncertainty during the process of training. In this section, we will give a detailed description of the proposed VI-PINNs.

Architecture of VI-PINNs
Vanilla PINNs put together the PDE, boundary conditions, and initial conditions into the total loss function, and the weights and biases are trained using certain optimization algorithms.
VI-PINNs is an extension of vanilla PINNs. We want to represent the model uncertainty directly, so we use a heteroscedastic neural network to output the predicted mean µ(x) and the variance σ 2 (x) in the final layer. Notably, in order to enforce variance to obey the positivity constraint, following [17], we pass the second output σ 2 (x) through the softplus function log(1 + exp(·)), and add a very small constant 10 −6 to the variance for numerical stability. The structure of VI-PINNs is shown in Figure 2.
Having decided the particular network architecture, how to combine the predicted mean µ(x) and the variance σ 2 (x) needs to be addressed. Naturally, we can minimize the negative log-likelihood criterion by considering a Gaussian assumption on target distribution, as described in Section 3.2. Mean square error and negative log-likelihood both have their advantages on regression problems, therefore we would like to incorporate them. Considering the successful application of mean square error in PINNs, we figure out a solution that negative log-likelihood can be used to assist the training procedure of mean square error, which is the auxiliary task mentioned in Section 3.3.

NLL loss for VI-PINNs
For regression problems, mean square error (MSE) is always taken as the loss function for optimizing. However, MSE alone does not have the ability to capture uncertainty. In order to capture uncertainty, we view the observed If we take the natural log of both sides, we will get the following form: Accordingly, negative log-likelihood loss (NLL) is taken for better optimizing model parameters over all x i : The last term is a constant and can be ignored for optimization, the simplified form of Equation (12) is: However, the NLL loss above does not perform satisfactorily well in our experiments. In our opinion, as the training process goes on, the variance σ 2 (x) becomes smaller and smaller, log σ 2 (x) becomes a big negative value, which might not be helpful to the training. So we adjust the NLL loss slightly, and the log σ 2 (x) will not be a negative value even if σ 2 (x) << 1, the modified negative log-likelihood(MNLL) loss is represented as the following form: Applying the loss function to Equation (1)- (3), we have the following form of L r (θ), L b (θ), and L 0 (θ) accordingly: The loss L (θ) is composed by loss term related to variance L r (θ), L b (θ) and L 0 (θ):

Auxiliary task in VI-PINNs
All machine learning tasks have certain goals. By introducing some related tasks with similar goals, the model can learn more useful information to improve the performance of the tasks we are concerned about, and these related tasks are called auxiliary tasks. Taking [40] as an example, the authors chose SIDE and semantic segmentation as the main tasks. By adding auxiliary tasks such as regression for the current time of day and classification of weather conditions, the model is able to learn a rich and robust common representation of an image, resulting in improved performance on the main tasks. It has been shown that auxiliary tasks can improve the performance, robustness, and training speed of networks. Instead of using the L (θ) solely to obtain the optimal model parameters, we prefer to introduce it to the vanilla PINNs loss function as an auxiliary task. For one thing, existing studies on PINNs have proved that the loss function proposed in [1] is simple but sufficient to represent what we want to learn from data to some extent, there is no need to utilize a totally novel loss function; for another, [40][41][42][43] conveyed to us that by introducing auxiliary tasks as complementary knowledge, the performance of the main task is expected to boost.
So we take the following loss function as the overall loss function of our VI-PINNs: where L(θ) and L (θ) are defined in detail in Equation (5) and (18) respectively, the weight λ is a hyperparameter needs to be adjusted manually to better assist the training process under different weights and bias initialization.

Results
In this section, we provide some numerical examples to demonstrate the performance of our proposed VI-PINNs against some other methods. The methods we compared are summarized in Table 1. In all examples, the activation function is tanh(·), a differentiable nonlinear activation function. The optimizer is Adam [35] with a fixed learning rate 0.001. All codes are implemented in Pytorch [44], a framework used widely in machine learning. In all cases below, we use fully connected networks containing three hidden layers, each with 30 neurons. In our experiments, we use full batch training rather than mini-batch training for our architecture is very simple. We train networks on a small dataset, and we test the performance on a larger dataset as the more points we use for testing, the more accurate the results will be. Mean square error (MSE) and L 2 -error are used as the metric for testing the performance of the trained model.
where u is the reference solution,û is the predicted solution.
To get rid of the effect of randomness in PINNs, we run the same experiment five times on the same dataset but with different random seeds of initialization to ensure that the results can be reproduced, and the mean errors of these five times are used to represent final errors. In the last subsection below, we also perform simple systematic studies of a simple equation to quantify predictive accuracy for different choices of the number of points, different network architectures, and hyperparameter λ.

Advection Equation
The first example we consider is the advection equation, with the initial condition, and boundary conditions, where the exact solution for this advection equation is, Compared to the other equations, the advection equation is easy to fit, so the total number of iterations is set to 8,000, and the number of collocation points, initial points, and boundary points are set to 500, 100, and 100, respectively. The performance of different methods is evaluated on a large dataset with 5,000 points. Table 2 displays MSE and L 2 -error of all methods and Figure 3 represents the MSE and L 2 -error values in iteration. As shown in Table 2, our proposed method gets the best result within five methods, even the worst performance can achieve an L 2 -error of 6.17e-04, while results of other methods are all bigger than 1e-03. As shown in Figure 3, our proposed method can get more accurate results while taking fewer training steps, while other methods get results that are close to each other except M3.

Burgers Equation
The Burgers equation considered here is defined as, Unlike other equations having a smooth solution, the solution of the burgers equation has a sharp internal layer. The total number of iterations is set to 15,000 and the number of collocation points, initial points, and boundary points is set to 5000, 500, and 500, respectively. We test the performance of different methods on 25,600 points used in [1]. Table 3 displays MSE and L 2error of all methods and Figure 4 represents the MSE and L 2 -error values in iteration. Figure 5 represents the exact solution and the best absolute error of different methods for the burgers equation.
As shown in Table 3, our proposed method gets the best performance, M2 also performs well, that is to say, for the burgers equation, emphasizing the effect of initial condition points helps to improve the performance as well. From Figure 4 we can conclude that as the training process goes on, our method converges faster than other methods, and M1, M3, and M4 hardly decline later. The solution of the burgers equation has a very steep domain when x = 0. Figure 5 shows that M1, M2, and M5 are all able to capture this feature, but our result fits the feature more closely compared to the other methods.

Convection-Diffusion Equation
Convection-diffusion equation is a linear PDE, containing an advection term and a diffusion term. The equation is defined as follows: where the exact solution is: Here we take c = 4.0, µ = 0.05, L = 4, T = 1 adopted in [45], the total number of iterations is set to 20,000 and the number of collocation points, initial points, and boundary points are set to 1,000, 100, and 100, respectively. The performance of different methods is evaluated on a large dataset with 5,000 points. Table 4 displays MSE and L 2 -error of all methods and Figure 6 represents the MSE and L 2 -error values in iteration. Figure 7 depicts the exact solution and the best absolute error between the exact solution and predicted solutions of the convection-diffusion equation. As shown in Table 4, within these five methods, our proposed method significantly outperforms the others. The MSE and L 2 -error of M2 hardly decline, meaning that in this equation putting emphasis on the effect of initial conditions may wind up with the opposite effect. M5 gets a better result compared with M1, denoting that it is practical to adopt a multi-object optimization strategy. From Figure 7, we can get that our method is capable of giving a more exact solution over the whole domain in contrast to the remaining methods.

Poisson Equation
The general form of the poisson equation is: with boundary conditions: Here, we take L = 2 √ π and consider using a special exact solution u(x) = sin(x 2 ), as x moves away from 0, the solution becomes oscillating. Now, we can fabricate the f (x) by derivation, the specific form becomes: The total number of iterations is set to 10,000, 2,500 points are sampled randomly for collocation points and we just need two points for boundary conditions. The performance of different methods is evaluated on a large dataset with 5,000 points. We display the mean MSE and L 2 -error of five duplicate experiments in Table 5, and we also represent the contrast between the exact and the best predicted solutions in Figure 8. We do not take into consideration M4 in this equation since M4 balances the loss by utilizing the relative progress of the current time-step t and the previous time-step t − 1, which is practical theoretically but may exceed the computational range of computers for the loss of this equation may change drastically between two consecutive training epochs. As shown in Table 5, M3 achieves the largest error given the number of training epochs for which it fails to fit the solution, as demonstrated in Figure 8. Figure 8 depicts the exact and the predicted solutions for the poisson equation, our proposed method achieves the most accurate result among all the methods, we can also figure out that the neural networks approximate the solutions gradually, and our method converges faster than other methods.

Wave Equation
The wave equation to be discussed is defined as: with initial conditions: boundary conditions: u(t, −L) = u(t, L) = 0, (40) and the exact solution of this wave equation is: Here, we take T = 2.0, L = 4.0, so T and L in Equation (37)-(41) can be substituted. The total number of iterations is set to 15,000 and the number of collocation points, initial points, partial initial points, and boundary points is set to 2,000, 200, 200, and 200, respectively. The performance of different methods is evaluated on a larger dataset with 5,000 points. Table 6 displays MSE and L 2 -error of all methods and Figure 9 represents the MSE and L 2 -error values in iteration. Figure 10 depicts the exact solution of the wave equation and the best absolute error between the exact solution and predicted solutions. As shown in Figure 9, the methods achieve similar results at the early training epochs. As the training process goes on, the result of our method drops significantly, while the others drop slightly. In this example, M2, M3,  and M4 perform similarly. Figure 10 represents plainly the absolute error, our method achieves the smallest absolute error while other methods get larger errors across a large region of the domain.

Comparative studies
In this subsection, we perform a comparative study on the advection equation used in section 4.1 in order to figure out whether the choices of points, different network architectures, and hyperparameter λ have big influences on the performance. Table 7 summarizes the relationship between the point selection of initial points, boundary points N u and collocation points N f , while the architecture is 3 hidden layers with 30 neurons per layer, the λ is fixed to 1. By analyzing the table, we can deduce a general trend that given a fixed amount of collocation points N f , the prediction becomes more accurate as the number of initial and boundary points N u increases, and vice versa. This is easy to explain, as the number of points increases, the model is able to acquire more useful knowledge, leading to improved performance.  Table 8 shows the L 2 -error for different network architectures, while the number of initial points, boundary points, and collocation points is fixed to 100, 100, and 500, respectively, the λ is fixed to 1 too. We observe that as the number of layers and neurons increases, the prediction becomes more accurate since the model becomes more powerful to approximate complex functions. While using too many layers and neurons can also cause problems such as overfitting, where the model is complex enough and the information contained in the training set is insufficient to train all the neurons, resulting in the degradation of performance.  Table 9 represents the influence of the hyperparameter λ, while the number of initial points, boundary points, and collocation points is fixed to 100, 100, and 500, respectively, the architecture is 3 hidden layers with 30 neurons per layer. As expected, different λ has a different influence on the accuracy of prediction.

Conclusion
In this paper, we introduce Variance-Involved Physics-Informed Neural Networks, a novel framework considering the influence of variance. We use an explicit and efficient way to model uncertainty by viewing the observed value as a sample from a Gaussian distribution. And we introduce our idea as an auxiliary to assist vanilla PINNs to achieve a more accurate result at a faster speed of convergence. Five numerical examples are presented in order to investigate the performance of our proposed method. Our results show that VI-PINNs can improve the accuracy of PINNs and converge faster.

Declarations
Competing Interests The authors have no competing interests to declare that are relevant to the content of this article.

Data availability
The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.