{'fraction_non_alphanumeric': 0.02809028266066657, 'fraction_numerical': 0.006574321473773028, 'mean_word_length': 4.793536560526852, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Overcoming model simplifications when quantifying predictive uncertainty

It is generally accepted that all models are wrong -- the difficulty is determining which are useful. Here, a useful model is considered as one that is capable of combining data and expert knowledge, through an inversion or calibration process, to adequately characterize the uncertainty in predictions of interest. This paper derives conditions that specify which simplified models are useful and how they should be calibrated. To start, the notion of an optimal simplification is defined. This relates the model simplifications to the nature of the data and predictions, and determines when a standard probabilistic calibration scheme is capable of accurately characterizing uncertainty. Furthermore, two additional conditions are defined for suboptimal models that determine when the simplifications can be safely ignored. The first allows a suboptimally simplified model to be used in a way that replicates the performance of an optimal model. This is achieved through the judicial selection of a prior term for the calibration process that explicitly includes the nature of the data, predictions and modelling simplifications. The second considers the dependency structure between the predictions and the available data to gain insights into when the simplifications can be overcome by using the right calibration data. Furthermore, the derived conditions are related to the commonly used calibration schemes based on Tikhonov and subspace regularization. To allow concrete insights to be obtained, the analysis is performed under a linear expansion of the model equations and where the predictive uncertainty is characterized via second order moments only.

1. Introduction. This paper considers the problem of assessing uncertainty in a prediction made for a particular system based on the combination of specific measurement data and expert domain knowledge. The focus of this work is environmental systems, such as river basins and groundwater systems, however, the analysis is likely to be applicable to a much wider set of domains. Correctly addressing such environmental prediction problems is fundamental to ensuring these important system are managed appropriately and sustainabl. Probabilistic Bayesian methods provide a theoretically consistent set of rules to combine such site specific data and prior knowledge through the use of a system model. However, these methods often fail when naively applied to a model that does not capture the full complexity of the system.
A possible solution is the incorporation of more detail and structural variations in the system model such that more sources of uncertainty are included. However, it is important to admit that this modelling ad. infinitum is not a solution as unlimited resources are never available for detailed model construction and execution, and simplifications must be made, at least at some level.
However, deciding on what should be included in a model of a system and what may be ignored is not straightforward. For instance in groundwater hydrology there is no consensus on what is an appropriate level of parameterization detail, e.g within the hydraulic conductivity field [24,48,47,51]. In addition to this are the related decisions of what processes should be explicitly represented and what can be ignored. Or alternatively, when is it reasonable to lump many different processes together under a semi-physical or non-physical black box model that directly represents input-output relationships [17,46].
What is necessary is an understanding of the effects of simplifying assumptions made during model development, and appropriate ways of dealing with them when calibrating the model and generating predictions.
A general, qualitative, notion of model adequacy was explored by [23] in terms of the issues faced within the surface water, groundwater, unsaturated zone, and terrestrial hydrome-teorology modelling communities. The work considered intermediate stages within the modelling process from the initial perceptual understanding through to the construction of a computational model and introduced a pluralistic definition of model adequacy. On one extreme is the engineering viewpoint that defines a structurally adequate model as one that can reproduce the input-output relationship of the system, with well characterized uncertainties (error models). On the other extreme is the physical science viewpoint that requires an adequate model to be consistent with the underlying physical system [23]. This paper will consider systems that may be exposed to future disturbances and limited data is available. Within these problems a well characterized regression model (the engineering viewpoint) cannot be constructed and some physical insight of the system is required [23]. Furthermore, focus is given to environmental management problems, where the role of a model is to help inform a subsequent decision problem related to risk management e.g. engineering design [18], groundwater management [15], or climate change [40]. This requires the specification of a probability distribution over the predictions of interest such that decision theoretic methods can be applied to quantify the risk and determine the optimal management strategy [5,43].
1.1. Related Work. The issue of mismatch between reality and a numerical representation of a system has received considerable attention from may different perspectives. The presence of simplifications may be identified in the data as additional misfit that is not consistent with measurements errors alone. These additional errors are referred to as: model error [30,51,29], model structural error [6], structural noise [16], model inadequacy [25,23], model discrepancy [19,44], modelization uncertainties [45], and others.
Within the system identification and control community [27,28] the concept is often referred to as system under-modelling [33,37] and two main probabilistic approaches have been developed: stochastic embedding [22,21,29] and model error modelling [26]. These allow subjective information about the discrepancy between the transfer functions that describe the dynamics of the real system and that of a numerical model to be explicitly defined and combined with a time series dataset collected from the system. This in turn allows prediction uncertainty to include uncertainty due to modelling errors and measurement errors contained within the data. These approaches generally allow a very flexible black box representation of the system dynamics to be used.
In the statistical modelling community, computational simulators have been considered as an explicit approximation to the real physical system they are attempting to represent. The mismatch is represented as an additional error term and modelled probabilistically, generally with explicit space time correlation [25,13,19,20,39,40]. Due to the computational complexity of the methods for high dimensional models, approximate Bayesian methods have also been developed [20]. Such simulators may also be decomposed and internal error terms included to represent the structural errors within smaller submodels [44].
Other work within hydrology has examined model simplifications and how these effect the calibration and prediction processes. In particular it has been shown that model parameters that are designed to represent particular physical properties may be forced to undertake surrogate roles to compensate for the simplifications during the model calibration process. This surrogacy has the potential to introduce additional biases into the predictions that are unaccounted for by typical probabilistic methods. This has been analyzed by explicitly modelling the simplification process in [31,16,14,50,51].
Furthermore, several strategies have been considered to overcome the approximations inherent in a simplified model. This includes the generalized likelihood uncertainty estimation (GLUE) method [7] that modifies the data likelihood function such that less information is extracted from the data. For a dynamical system, the assumption of a deterministic dy-namic transition function can be relaxed and a stochastic process or transition function used instead [49,11] (such models are often referred to as data assimilation methods). Similarly the assumption that the model parameters are time invariant may be related such that they can change over time to better match the observed system behavior [36].
It is noted that many of these methods rely on a noticeable discrepancy between the measured data and model predicted values. However, the adverse effects of model simplifications may not necessarily cause any additional misfit between the data and model output, and thus may go unnoticed during model calibration and prediction [51]. Thus it is critical to have a theoretical understanding of model adequacy that goes beyond data fit.
1.2. Approach and Contributions. This paper considers a subjective Bayesian framework and formally defines what is an adequate or appropriate model by explicitly considering: (i) the nature of the predictions, (ii) the available data and prior knowledge, (iii) the model simplification strategy, and (iv) the calibration scheme. It is shown that these are intrinsically linked and performing model calibration should take into account the nature of the data, predictions and any inherent limitations of the simplified computational model. It is noted that this departs from classical guidance on calibration and inversion that focuses on data misfit only [30,45,2].
A dual model approach is used, where a "reference" [31] or "reality" [51] model is used to describe how the system is believed to function. Such a high fidelity reference model has been used in the past to characterize the performance of a simplified model. Here, the approach is extended to explicitly determine when and how a simplified model can be used to generate an accurate, or at least conservative, estimate of uncertainty in a prediction of interest. To allow concrete insights to be produced, only linear(ized) problems are considered and the beliefs are restricted to be Gaussian, where the second order moment is sufficient to capture uncertainty.
Within this framework, model simplifications are represented as a subspace projection that restricts the flexibility of the simplified model. This prevents the model from being able to fully represent the complexity of how the real system is believed to function. It is shown that if the model simplifications are ignored and standard probabilistic calibration methods are used, the model may generate overconfident and non-conservative predictions. Generally, this is avoided only when the simplification strategy is optimal. The key contribution of this paper is the characterization of two new calibration and prediction schemes for suboptimally simplified models that avoid this under estimation of uncertainty. The first scheme allows a simplified model to be used in a way that replicates the performance of an optimal scheme through the appropriate modification of the prior, or regularization, term. In the second scheme, focus is given on linking the predictions with the available data such that insight is gained into when the simplifications can be overcome by gathering the appropriate set of data.
Specifically, this paper fully defines the general problem of assessing uncertainty with a simplified model in Section 2. Section 3 summarizes the optimal benchmark solution based on the high fidelity reference model. The general representation of model simplifications based on linear projection is defined in Section 4. A typical probabilistic calibration scheme is considered in Section 5 that ignores the effects of model simplifications within the calibration and prediction process. This section defines the concept of an optimal simplification that determines when such a scheme is adequate for assessing uncertainty. Section 6 considers suboptimally simplified models and introduces two new calibration and prediction schemes, and defines the conditions when they are adequate for assessing predictive uncertainty. Discussions and directions for future research are covered in Section 7. A worked example involving a simplified groundwater prediction problem is given in Appendix A.
2. Problem Definition. Consider a system where a modeller is tasked with generating a probability distribution over a prediction of a specific feature of the system to inform a future decision process. For instance in a groundwater system, the prediction of interest may be the reduction in water levels that will be experienced by an aquifer due to an increase in future extractions. The prediction of interest will be denoted by the vector p ∈ R Dp .
To aid the modeller, a limited number of measurements have been made on the system and will be represented by d ∈ R D d . Furthermore, it is considered that expert prior knowledge exists on how the system functions, for instance the physical processes involved, and allows the available data and prediction of interest to be related to the properties of the underlying system. This information shall be considered to define a high fidelity or reference model for the system with parameters denoted by the vector x ∈ R Dx . Furthermore, the relationship between the data d, predictions p, and the underlying parameters x that the modeller believes exists shall be represented by the pair of equations It is considered that these functions include the operation of known physical laws, such as conservation of mass, energy and momentum, while the vector x includes a detailed representation of the forcing terms, initial conditions, material properties, etc. These requirements result in a very high dimensional vector x and complex functions G and Y.
Furthermore, it is considered that epistemic uncertainty exists as to the value of x for the system under investigation. In addition, the errors on the right of (1) and (2) allow for the inclusion of additional uncertainty, for instance to capture measurement errors introduced by the data acquisition method, or uncertainty that may exist in how a physical process actually functions [45].
This allows the modeller's complete belief about how the system behaves to be captured by three probability distribution functions (3) p(x), p(d|x), p(p|x).
These in turn can be used to generate a posterior distribution over the predictions of interest, denoted as p(p|d), using standard rules of probability theory. It is noted that a major issue with pursuing this type of subjective Bayesian approach is the requirement that these probability distributions must represent every detail that is believed to be present in the physical system. This is not possible and simplification must be made.

Simplifications.
It is considered that the process of modelling, for instance as outlined in [23], transforms the detailed knowledge of the modeller and produces the desired simplified computational representation of the problem that can be solved within the typical computational constraints. Specifically, the output of the modelling process is (or should be) an approximate representation of the modeller's beliefs that can be processed numerically to generate a prediction posterior distribution that is somehow similar to the optimal, but computationally intractable, distribution p(p|d) generated by the high fidelity model. The modelling process is depicted in Figure 1 where the outputs are denoted by the set of approximate beliefs (4)p(v),p(d|v),p(p|v).
These probability distributions are defined over a simplified description of the system, denoted by the parameter vector v ∈ R Dv . Furthermore, it is considered that the approximate Simplified beliefs: The process of modelling is to transform the modeller's beliefs into a set of approximate probabilities that can be processed numerically to generate a conservative posterior probability distribution over the predictions of interest. posteriorp(p|d) should not just be similar to posterior p(p|d), it should be in some sense conservative such that the uncertainty is not underestimated [15].
Thus, the key question that is addressed in this paper is now specified as: How can the subjective beliefs of a modeller, defined for the high fidelity reference model, be transformed to generate approximate probabilities that define a simplified computational model such that the computed prediction posteriorp(p|d) is a conservative approximation of the optimal reference posterior p(p|d)?
This question explicitly links the typically separate problems of: model construction, calibration and prediction.
Furthermore, answers will be provided under the restriction that the uncertainties are described by multivariate Gaussian distributions and the dependencies are linear. This will allow more specific and concrete insights to be gained into the effects of simplifications and how they may be overcome. Extensions to more general non-Gaussian and nonlinear systems is left for future work.
3. Optimal Inference and Prediction. The optimal Bayesian prediction scheme, which does not consider any simplifications, is now briefly reviewed [5,45,2]. This is based on the high fidelity reference model.
Let the predictions of interest, the available data, and the parameters of the high fidelity model be defined by the random vectors: p ∈ R Dp , d ∈ R D d , and x ∈ R Dx , where D p , D d and D x are their respective dimensions. It is considered that the dimension of x is excessively large, while the data is limited, and the number of predictions is small (perhaps only one), such that D x D d , D p . Prior knowledge of the parameter vector is represented by the Gaussian probability density function p(x), with mean of zero and known second order moment Σ x such that It is noted that the requirement of a zero mean distribution simplifies the analysis and can be met in general with a transformation into increments. Now, the believed relationship between the system properties x, and the data vector d is considered to be linear with coefficient matrix G. This can be considered as a linearized approximation of (1), however the approximation introduced by the linearization is considered out of scope in this paper. In addition, the uncertainty that is believed to exist in this relation-ship is represented by the error term δ such that d = Gx + δ. Furthermore, the uncertainty in the value of δ is represented by a zero mean Gaussian density with known covariance matrix of Σ δ . In the simplest case, the matrix Σ δ represents errors within the measurement process. This allows the modeller's conditional belief of the measured data given the underlying parameters to be defined by the Gaussian density (6) p(d|x) = N (d; Gx, Σ δ ).
The prediction of interest p is also considered to be linearly dependent on x with a coefficient matrix Y, and an additive error denoted by ρ, such that p = Yx + ρ. In addition, it is considered that the uncertainty in the value of the error is a zero mean Gaussian density with known covariance Σ ρ . It is noted that this requires that δ and ρ are independent. If any correlation exists, it must be included in x. Finally, the conditional belief of the prediction given the system properties can now be defined as (7) p(p|x) = N (p; Yx, Σ ρ ).
Note that if the system is believed to be well described by a deterministic model, the covariance of the prediction error could be considered negligible. The calibration or inversion stage seeks to determine the posterior belief over the parameters x given the available data d Here, the posterior density p(x|d) = N (x; µ x|d , Σ x|d ) is Gaussian with a mean and covariance given by The coefficient matrix of the data d in (8) is sometimes referred to as the optimal estimator, or gain matrix, and will be denoted by To propagate the posterior belief into the prediction of interest, requires the integration over the underlying parameters p(p|d) = p(p|x)p(x|d)dx.
The prediction posterior density p(p|d) = N (p; µ p|d , Σ p|d ) is Gaussian with mean and covariance given by The overall calibration and prediction scheme is now summarized below. This sets the benchmark for the other schemes considered in this paper. DEFINITION 1 (Optimal Scheme). The optimal scheme is denoted by the set of linear Gaussian probability density functions p(x), p(d|x) and p(p|x), parameterized by the matrices Σ x , G, Σ δ , Y, and Σ ρ , as defined in (5)-(7).
Bayesian network depicting the structure of the densities p(x), p(d|x), p(p|x) that employ the high fidelity model. For a given dataset d, the posterior belief over the predictions of interest is the Gaussian density p(p|d) = N (p; µ p|d , Σ p|d ), with mean and covariance defined by the functions µ(·) and Σ (·) respectively Note that the covariance matrix of the posterior is not a function of the data d.
The structure of the above prediction scheme that encapsulates the prior densities p(x), p(d|x), p(p|x) that are based on the high fidelity model is depicted graphically in Figure 2 4. Model Simplifications. The simplified computational model is considered to contain an approximate system simulator, with exposed parameters v and linear structure analogous to that defined for the high fidelity model p ≈Ỹv + ρ.
HereG andỸ are linear data and prediction matrices of a simplified system simulator. The exact form of the approximations within these relationships are of critical interest and will be explored by explicitly representing the simplifications involved.
To achieve this consider initially that the parameters v are somehow physically inspired and have some meaning to the modeller in describing aspects of the system. This allows the parameters of the simplified model to be used to describe a restricted version of the high fidelity model with parameters x ∈ R Dx determined by the parameter vector of the simplified model v ∈ R Dv via a matrix C ∈ R Dx×Dv (14) x = Cv .
This explicit representation of model simplifications allows aspects of the system to be ignored [16,51] and allows for the incorporation of arbitrary linear parameterization schemes, such as spatial and temporal homogeneity [30]. With this definition, the simplified data and prediction matrices may be rewritten in terms of the matrices of the high fidelity modelG The matrix C will be referred to as a simplification matrix and provides a way of linking the simplified simulator to the original high fidelity reference model of how the system functions.
Finally, it is important to note that physical intuition is not strictly required when defining the simple model as for any G, Y andG,Ỹ, such that G Y has full row rank, there is always a C that satisfies (15) and (16). However, this may not be unique and thus there are potentially many different ways to interpret the meaning of the simple model. Here, it is considered that the modeller specifies the interpretation by e.g. specifying C directly.

Representing Unmodelled Complexity.
A simplification matrix C ∈ R Dx×Dv , divides the space R Dx into two perpendicular subspaces: columnspace(C) and cokernel(C). The column space contains the set of parameter vectors that can be explicitly represented by a low dimensional vector. The second subspace cokernel(C) contains vectors that includes some degree of complexity that cannot be represented by a low dimensional vector.
Before defining these further, the following assumption is made.
It is noted that if a simplification matrix C does not have full column rank, then there will be parameter combinations of the simple model that have the same influence on the high fidelity model, which implies that the simple model is not as simple as it could be for the same expressive power. Now, consider the singular value decomposition of C under the above assumption where U C is the collection of orthogonal unit vectors that span the column space of C and similarly UC spans the subspace perpendicular to this, the cokernel of C. This enables the parameter vector of the high fidelity model to be expanded as where v is the parameter vector of the simple model and u is a random vector that captures all the unmodelled complexity. Finally, it is noted that this expansion is unique and only dependent on the definition of the simplification matrix C.
Under the above expansion, the data and prediction is related to the two components v and u through From these equations, it is noted that the unmodelled components of the system will be expressed in the data whenever GUC = 0 and in the predictions whenever YUC = 0. These additional error terms are denoted as η and respectively and play a very different role to the other error terms δ and ρ as they are correlated via u. It is noted that the additional error η introduced by the model simplifications was explicitly considered in [30] and [45] and was used to define a composite measurement error model for the sum δ + η. However, the effects of the simplifications on the prediction, including the explicit correlation between η and , was not addressed.
In an ideal world it could be argued that the objective of any modelling exercise should be to construct a model that is a simplified version of how the system is believed to function that captures the full complexity of the available data and predictions of interest. That is, which has a C such that η = = 0. Such a simplification matrix always exists, and will be referred to as an optimal simplification. DEFINITION 2 (Optimal Simplification). Let G and Y be data and predictions matrices for a high fidelity system model. Then a simplification matrix C is optimal if where UC spans the cokernel of C, as defined in (17).
It is noted that the above definition of optimal simplification differs from that of [16,14,50] in that it explicitly considers the nature of the predictions. An optimal simplification matrix, C * , can be found for any given data and prediction matrices G and Y, by first taking a singular value decomposition of the matrix Now, an optimal simplification matrix is given by It is considered that an optimally simplified model is difficult, or even achievable, to obtain in practice and the remainder of the paper will focus on understanding and overcoming the issues caused by suboptimal simplifications.

Naive Use of Simplified Models.
Here, a prediction scheme is defined that first infers the parameters of the simple model and then propagates them into the prediction, but ignores the potential for unmodelled complexity to be expressed in the data or predictions. This represents a standard probabilistic Bayesian approach to model calibration and prediction. It is shown that this scheme is generally not conservative as the uncertainty in the predictions is underestimated.
5.1. Prior Information. The scheme will have the same basic structure as the optimal benchmark scheme. In particular it will include an explicit representation of a prior over the parametersp n (v), a data likelihoodp n (d|v) and conditional density for the prediction p n (p|v). Here the superscript n denotes approximate distributions associated with this naive method.
The conditional densitiesp n (d|v) andp n (p|v) are defined directly with the simplified data and prediction matricesG andỸ respectively. Furthermore, the covariances are set to those used in the optimal scheme i.e.
This formulation of the data and prediction densities is equivalent to ignoring the approximations in (12) and (13) introduced by the model simplifications. Now, to be somewhat rigorous, the specification of a prior distribution on the parameters v is performed by explicitly considering the nature of the simplification and the uncertainty in the parameters of the high fidelity model. This is accomplished by considering a transformation that propagates the prior distribution over the parameters of the high fidelity model x into the joint space formed by the parameter vector of the simple model v and the vector u that denotes the unmodelled complexity. This is defined in the following proposition. PROPOSITION 1 (Uncertainty Propagation). Let p(x) = N (x; 0; Σ x ) be a zero mean Gaussian density, with covariance Σ x , that captures the prior uncertainty in the parameters of a high fidelity model. Also, let C be a simplification matrix that describes a simplified model.
Then, the joint probability density function for v and u, representing the modelled and unmodelled components of the simple model respectively, is the zero mean correlated Gaussian density where the covariance terms are given by Here, (·) † is the pseudoinverse operator and UC spans the cokernel of C as defined in (17).
See Appendix B for a proof. This proposition defines the covariance matrix Σ v , that represents the uncertainty in the parameters of the simplified model, to be the orthogonal projection of Σ x onto the column space of C. This is the most rigorous approach to specifying the prior over the parameters of the simple model as it preserves the uncertainty that is believed to exist and is consistent with the physical intuition used to construct the simplified model. This marginal density is used to define the prior of the naive prediction schemê

Naive Calibration and Prediction.
The naive calibration and prediction scheme is now defined that generates a posterior via the standard rules of probability theory. This is similar to the optimal scheme, but without consideration to the simplifications within the model's data and prediction equations. DEFINITION 3 (Naive Scheme). A naive prediction scheme is denoted by the simplified linear Gaussian density functionsp n (v),p n (d|v) andp n (p|v) parameterized by the matrices Σ v ,G, Σ δ ,Ỹ, and Σ ρ .
Furthermore, the prior covariance matrix Σ v is defined in terms of a high fidelity model using Proposition 1, that is where Σ x denotes a covariance matrix that describes the uncertainty in the parameters of the high fidelity model, and C denotes the simplification matrix that links the parameters of the simple and high fidelity models. For a given dataset d, the posterior belief over the predictions of interest generated by this naive scheme is the Gaussian densityp n (p|d) = N (p;μ n p|d ,Σ n p|d ), with mean and covariance defined asμ where the functions µ(·) and Σ (·) are given in Definition 1.

Performance.
To assess the performance of the naive scheme the notion of conservativeness, discussed in Section 2, is first defined by considering the squared error in the mean prediction. This is performed for a pair of probability density functions in Definition 4, and extended to calibration and prediction schemes in Definition 5. A further generalization of these definitions, which explicitly incorporates the utility function of the subsequent decision problem, is proposed in Section 7. DEFINITION 4 (Conservative Density). Let p(p) andp(p) be density functions defined over the random vector p. Furthermore, letμ p be the mean ofp(p). Thenp(p) is defined as a conservative approximation of the reference density p(p) if the approximate density's expected mean squared error Ep (p) {(μ p − p)(μ p − p) }, does not increase when the expectation is instead taken with respect to reference density p(p) It is noted that condition (28) can be rewritten in terms of the means µ p ,μ p and covariances Σ p ,Σ p of the two densities, and yields the condition An example of a conservative density, that satisfies Definition 4, is given in Figure 3.
The notion of conservativeness is now generalized to a prediction scheme. This is performed by considering an expectation over typical data.
DEFINITION 5 (Conservative Scheme). Let the densities p(x), p(d|x), and p(p|x) denote the prior information of a reference scheme. Also, let p(p|d) denote the posterior density this scheme generates for a given dataset d. Furthermore, letp(p|d) denote a posterior density generated by an approximate scheme.
For the given dataset d, the degree of conservativeness of the approximate posterior is denoted by the function Now, the approximate scheme is defined as conservative if the expectation of the function Ω(d) is positive semi-definite Here, the density p(d) denotes how probable different datasets are under the prior knowledge of the reference scheme and is given by It is noted that for the linear Gaussian densities considered in this work, the covariance of the posterior distributions are independent of the data and (30) can be simplified as Thus, a scheme is considered as conservative if it generates a posterior covariance matrix Σ p|d that is inflated with respect to Σ p|d by an amount dependent on the average squared difference in the posterior means.
It is now of interest to consider the performance of the naive scheme and determine when it is conservative with respect to the optimal benchmark scheme. This is performed in the following proposition. PROPOSITION 2 (Performance of Naive Scheme). Suppose the matrices Σ x , G, Σ δ , Y, Σ ρ define an optimal calibration and prediction scheme, with posterior density denoted by p(p|d) = N (p; µ p|d , Σ p|d ) for arbitrary dataset d.
Let C denote a simplification matrix such that the matrices Σ v = C † Σ x C † ,G = GC, Σ δ ,Ỹ = YC, Σ ρ define a naive prediction scheme for a simplified model. Now, consider the posterior density generated by the naive schemep n (p|d) = N (p;μ n p|d ,Σ n p|d ). 1. If C is an optimal simplification, then the generated mean and covariance of the naive prediction scheme are equivalent to those of the optimal prediction scheme, that isμ n p|d = µ p|d andΣ n p|d = Σ p|d for all d.
2. If C is a suboptimal simplification, then the naive scheme is conservative, provided the following condition holds where E n = Σ vG (GΣ vG + Σ δ ) −1 denotes the estimator matrix of the naive scheme and UC spans the cokernel of C as defined in (17). 3. If C is a suboptimal simplification, condition (32) does not hold, u and v are considered independent, and the covariance in the unmodelled complexity is non-zero Σ u = 0, then the naive scheme is strictly non-conservative.
See Appendix C for a proof. Under a suboptimal simplification, the non-conservative nature of the prediction scheme is caused by the influence of unmodelled complexity on the data and/or predictions. If it influences the predictions, then there is a direct bias effect. If it influences the data, then this causes the estimated parameters of the simple model to become biased by forcing them to take on surrogate roles to compensate for the simplifications. Furthermore, this bias in the estimated parameters is propagated to the prediction. Neither of these effects are taken into account by the naive prediction scheme and the uncertainty is underestimated.
The condition in (32), which guarantees the scheme is conservative, requires that these two sources of errors in the prediction caused by the unmodelled complexity to exactly cancel each other out. This is unlikely to hold in practical scenarios without careful attention to the data, predictions, simplifications, and the type of prior knowledge available. This condition will form the basis of the data driven prediction scheme introduced in Section 6.
Bayesian network depicting the structure of the densitiesp n (v),p n (d|v), andp n (p|v) employed by the naive scheme. Also displayed in dashed lines is the dependency on unmodelled complexity u under the conditions of an optimal simplification. Note that under an optimal simplification, the unmodelled complexity does not have any direct effect on the data or predictions.
Finally, it is noted that for the general case when v and u are not independent and (32) does not hold, the naive scheme is still not guaranteed to be conservative. However there are more special cases which may produce conservative posterior densities. The conditions that delineate the strictly conservative from the non-conservative scenarios are likely to be of limited interest in realistic problems and have not been enumerate here.

Summary.
This section has formally defined a probabilistic calibration and prediction scheme that embeds the standard separation of modelling, calibration and prediction. In particular, physical knowledge is used to define a prior probability distribution for the parameters included in the simplified model. Also, the conditional data and prediction probability distributions are defined by ignoring the simplifying assumptions used to construct the data and prediction equations. Furthermore, it is shown that a true characterization of predictive uncertainty typically requires an optimally simplified model. For suboptimally simplified models it is shown that this scheme generally underestimates the uncertainty in the predictions and is not conservative.
The dependency structure of the probability distributions for this naive scheme is depicted in Figure 4. Also displayed is the dependency on the unmodelled complexity for an optimally simplified model. It is noted that under this condition the data and predictions are conditionally independent of the unmodelled complexity u given the model parameters v.
Finally, it is noted that the non-conservative, or overconfident, nature of predictions generated by naively applying probabilistic Bayesian methods on simplified models is not a new finding. In particular, the result can be considered as a generalization of [16,14,51] and is consistent with arguments of [7].
6. Overcoming Model Simplifications. It is of interest now to understand what to do about the issue of overconfidence when a suboptimal model is used. Firstly, it is noted that this issue may not be a problem. For instance, the prediction uncertainty may turn out to be larger than expected, and this in itself may provide sufficient information to allow a given management decision to be made. However, the situation becomes more difficult when it is necessary to determine an accurate or at least conservative prediction probability distribution, for instance as required by [15]. In this scenario the modeller may opt to: • Expand the sources of uncertainty considered in the model through less simplistic modelling assumptions with the hope of producing an optimally simplified model. This is not just about representing greater spatial/temporal detail in the model [24], but more importantly the goal should be the inclusion of more sources of uncertainty. As an example, if there exists uncertainty over the presence of some important structural feature, a multi-hypothesis, or multi-model, analysis may be needed [34,35,38]. • Directly model the additional errors introduced by the model simplifications. For instance by defining an additional probabilistic model for p(η, ) introduced in (19) and (20) that captures the uncertainty the modeller has in how well the simplistic model represents the behavior of the system. Such approaches have been developed in [25,13,19,20,39,40]. • Within the context of a dynamical system, remove the deterministic assumption on how the system evolves over time. This allows the simplifications to be represented by an error model within the modelled transition function. Such a scheme is employed by [49]. Alternatively, the assumption that the model parameters are time invariant can be relaxed such that the parameters can change over time to better match the underlying system behavior and inject greater uncertainty in the predictions [36]. • Modify the way that the data is used through changing the likelihood model p (d|v) such that less information is propagated into the prediction via the model parameters.
The generalized likelihood uncertainty estimation method [7] can be considered as an example of this approach. In the remainder of this section, two additional approaches are defined, along with explicit conditions that determine when they are appropriate. Firstly a scheme is defined that allows the posterior of the ideal prediction scheme to be reproduced using a suboptimally simplified model through the appropriate modifications of the prior covariances. Secondly, structural considerations of the data and predictions are considered and an approach developed that allows the simplifications to be overcome through the use of the right calibration data.
6.1. Optimal Use of Simplified Models. The naive scheme developed previously was not conservative for a suboptimally simplified model. Here a prediction scheme is defined that is capable of reproducing the optimal scheme, even under suboptimal simplifications.
To start, consider the probability densitiesp o (v),p o (d|v),p o (p|v) to be defined in a similar manner to the naive scheme, but parameterized in terms of new covariance matriceŝ Here the covariance matrices will be considered as adjustable such that they can be chosen in such a way that they compensate for the simplifications and allow the optimal posterior to be reproduced. Thus, it is of interest to selectΣ o v ,Σ o δ andΣ o ρ such that the posterior density that is produced by their combination,p o (p|d), replicates the posterior of the optimal scheme, i.e. The posterior densityp o (p|d) is Gaussian, with mean and covariance defined in a similar fashion to the optimal scheme. Thus, the above condition will be satisfied when the means and covariances are equivalent, which occurs wheñ If these conditions can be met, it defines a set of optimal covariance matrices of the simplified probability density functions such that when they are used to generate a posterior, the performance of the optimal scheme is replicated.
6.1.1. Highly Parameterized Models. The above conditions will now be specialized for a highly parameterized model, where the calibration problem is under-constrained. Such models are recommended by [24] and [51], and the results will explicit determine when such approaches are appropriate. This will be performed by focusing attention only on the prior covariance matrix for the parameters of the simple model, such thatΣ o δ andΣ o ρ remain the same as those used in the naive and optimal schemes, i.e.Σ o δ = Σ δ andΣ o ρ = Σ ρ . Such a scheme is now defined.
This scheme is now shown to be equivalent to the optimal scheme when applied to a suboptimal but highly parameterized model where the number of free parameters is at least as large as the number of linearly independent measurements and predictions of interest. PROPOSITION 3 (Performance of Compensated Scheme). Suppose the matrices Σ x , G, Σ δ , Y, Σ ρ define an optimal calibration and prediction scheme, with posterior density denoted by p(p|d) = N (p; µ p|d , Σ p|d ) for arbitrary dataset d.
Then, the posterior mean and covariance generated by the optimally compensated scheme are equivalent to those of the optimal schemê See Appendix D for a proof. This demonstrates that given a model simplified in a suboptimal fashion, the optimal performance can be recovered through the adjustment of the assumed prior uncertainties. The main rank condition in (38) is fairly easy to achieve in practice and is met when, e.g. the number of parameters is not smaller than the number of linearly independent measurements and predictions, and the simplified model matrixZ = ZC has full (row) rank.
A model may be highly parameterized due to the degrees of spatial variability in e.g. the modelled material properties that is included in the model (as in [51]). However, the above result can apply to models of dynamical systems that use stochastic transition models (sometimes referred to as data assimilation methods). In these models the additional dynamic error terms that are incorporated at each time increment can be similarly viewed as a set of additional model parameters.
The general structure of the optimally compensated calibration and prediction scheme is depicted with a Bayesian network in Figure 5. It is noted that the difference between this scheme and the naive scheme lies only in how the prior covariance matrix for the parameters of the simple model is specified. They are both defined as a transformation of the covariance matrix Σ x that captures the uncertainty in the parameters for the high fidelity model. Recall that the naive scheme uses while the optimally compensated scheme useŝ The matrix R is dependent on the simplification matrix C but it is also dependent on the data and predictions matrices G and Y of the high fidelity models. It is this explicit dependency on the data and predictions that allows the prior covariance matrixΣ o v to compensate for the errors in the data and prediction equations of the simplified model.
To gain additional intuition as to how the compensating prior covariance matrixΣ o v is related to Σ v used by the naive scheme, consider for the moment that v and u are independent. This enables the prior covariance matrix for the parameters of the high fidelity model to be decomposed as Σ x = CΣ v C + UCΣ u U C . With this, the prior matrixΣ o v can be rewritten as where Σ + =Z † ZUCΣ u U C Z Z † and is positive semi-definite matrix. This means that forΣ o v to compensate for the simplifications, the parameters must be allowed greater flexibility than would be given by the naive use of Note that under such a simplification, the unmodelled complexity has a direct effect on the data and/or predictions.

Weighted Least Squares Inversion.
In addition to the Bayesian arguments provided above, now consider a regularized weighted least squares formulation commonly used for calibration and inversion problems [30,45]. With the assumption of linear models, the mean vectorμ o v|d is also the maximum a posteriori estimate of the posterior parameter distribution of the simplified modelp o (v|d) ∝p o (d|v)p o (v), and may be equivalently defined as the solution to the regularized weighted least squares optimization problem (40) Note that this explicitly employs the simplified forward modelGv. Furthermore, the prediction generated by this estimateμ o p|d =Ỹμ o v|d is the mean and mode of the prediction posteriorp o (p|d). Due to the optimality of the compensated scheme, this point prediction is also the optimal minimum error variance prediction that the high fidelity model would generate.
This demonstrates that a simplified, but highly parameterized model, can be calibrated through the careful selection of a Tikhonov regularizer to generate the optimal prediction. Additionally, this optimal regularizer can also be used to characterize the predictive uncertainty of this point prediction through the use of the covariance formula in (37). 6.1.3. Summary. The introduced optimally compensated prediction scheme allows suboptimal model simplifications to be overcome through modifying the prior probability distribution for the parameters of the simplified model. This explicitly links: (i) the modelling problem, through the selection of a simplification matrix C; (ii) the nature of the available data via G; (iii) the predictions of interest defined by Y; and (iv) how the simple model is calibrated, as the required regularization termΣ o v is explicitly dependent on all three terms. It is noted however, that this explicit dependency on the high fidelity model will make the generation ofΣ o v problematic in practice as the matrix G and Y will not generally be available. Nevertheless, it formally defines how a prior regularization term for a suboptimal model should be selected.

Data Driven Predictions.
It was noted that the main barrier to applying the optimally compensated scheme defined above is the difficulty in determining the covariance  Here a new scheme is proposed that overcomes this issue by moving away from optimality and focusing on conservativeness.
In particular, a specific class of predictions will be considered that allow the simplifications to be hidden behind the data. Thus, the objective is to examine how model simplifications can be overcome through the collection of the right data. The intuition here is that data of a similar type to the predictions is often available, for example groundwater head measurements are often available when it is of interest to predict heads, similarly stream flow measurements are often available when predictions of stream flows are of interest, etc.
The class of predictions that will be considered here incorporates two important principles: • The effects of all unmodelled components of the system on the predictions are captured by the data. • Any component of the system that effects the predictions, and is not captured by the data, is explicitly represented in the simplified model. From these principles it is considered that the prediction of interest can be explicitly decomposed into an intermediate data term that is dependent on g = Gx and a term wholly dependent on the parameters of the simple model, that is whereĀ andB are arbitrary matrices. Predictions with this structure are conditionally independent of u given g and v, i.e. p(p|g, v, u) = p(p|g, v). This structure is represented by the Bayesian network in Figure 6 and requires the prediction matrix to have the following form (42) Y =ĀG +BC † .
It will be shown in the sequel that even with this restricted form it is not possible to overcome the simplifications and further constraints must be imposed.
6.2.1. Naive Scheme. Now, consider the naive calibration and prediction scheme introduced in Definition 3. Recall that this method will be conservative for a suboptimal simplification when condition (32), introduced in Proposition 2, is satisfied. Specifically, this requires YUC −ỸE n GUC = 0, where E n = Σ vG (GΣ vG + Σ δ ) −1 . For a prediction matrix Y with structure consistent with (42), this condition can be rewritten as Condition (43a) requires the random vectorb =Bv to be uncorrelated withg =Gv under the prior covariance Σ v . Furthermore, condition (43b) requiresGΣ vG Σ δ and occurs when the data is perfect such that Σ δ ≈ 0, or when the prior knowledge in the subspace that is informed by the data (i.e. rowspace(G)) is very weak such thatGΣ vG ≈ ∞.
These conditions ensure that: (i) the subset of parameters of the simple model that directly influence the predictions, represented byb =Bv, cannot be estimated from the data and (ii) the simple model can exactly reproduce the data.
It is important to note however that these conditions are unlikely to hold in practical scenarios and the naive scheme is not guaranteed to be conservative, even for the predictions with the structure considered in (41).

Data Driven Scheme.
To overcome the above issues, it is proposed to ensure conservativeness through the judicial removal of information such that the conditions defined in (43) can be satisfied. This will occur in two main areas: • An easily computable inflation of the prior covariance matrix Σ v .
• The application of a preprocessing or filtering step that may discard or combine some components of the data. Additionally, the class of predictions must be restricted further than those allowed by (41).
To start, let F ∈ R D d ×D d denote a data filtering matrix such that Similarly define the transformed data matrices as G = FG, andG = FG, and the transformed data error covariance as Σ δ = FΣ δ F . The purpose of the filter F will be elaborated on later. Note however, filtering is optional and the identity matrix can be used F = I. Now, consider the singular value decomposition ofG The last expression U 1 S 1 V 1 only includes the nonzero singular values, and will be referred to as the compact SVD. With this expansion, the vectors v 1 = V 1 v and v 2 = V 2 v define the rowspace and nullspace components of the parameters of the simple model for the given filtered data matrixG = FG. To meet condition (43b), the prior covariance will be inflated such that all information pertaining to the rowspace ofG , i.e. the subspace spanned by V 1 , is removed. In addition any correlation in the random vectors v 1 and v 2 will be ignored. Such a calibration and prediction scheme that embodies these properties is now defined.

a data driven scheme is denoted by the set of simplified linear Gaussian probability density functionsp
To fully define the prior covariance matrixΣ d v , let the covariance matrix Σ v denote the uncertainty in the parameters of the simplified model defined in Proposition 1 as Σ v = C † Σ x C † . Furthermore, let the matrices V 1 and V 2 each denote a set of orthonormal column vectors that form a basis for the rowspace and nullspace ofG , e.g. as defined in (45). The prior covariance matrixΣ d v is now defined by the limit For a given dataset d the posterior belief over the predictions of interest generated by the data driven scheme is the Gaussian densityp d (p|d) = N (p;μ d p|d ,Σ d p|d ), with mean and covariance given byμ where the functions µ(·) and Σ (·) are given in Definition 1.
Finally, a more specific class of predictions is considered than that defined in (42), such that condition (43a) will hold by construction. Specifically, the prediction must depend on the uncorrupted version of the filtered data, denoted by g = FGx, or on the components of the parameters of the simple model that line in the nullspace ofG , denoted by v 2 = V 2 v. The prediction cannot depend on any rowspace components directly. Predictions with this structure can be written in the form where A and B are arbitrary matrices. Furthermore, the dependency structure of this restricted class of predictions is depicted in Figure 7. The key difference between this and the structure depicted in Figure 6 is the explicit separation of v into v 1 and v 2 and the restriction that v 1 cannot have a direct dependency on the prediction. That is, the prediction p is conditionally independent of u and v 1 given g and v 2 , or equivalently p(p|g , v 1 , v 2 , u) = p(p|g , v 2 ). A prediction with this restricted structure has a prediction matrix of the form It is noted that the matrices A and B are arbitrary, the important characteristic is the structure that links the data and predictions to the simplification. This structure not only ensures that the simplifications are hidden behind the data, but also that any surrogate roles that the parameter vector v 1 is forced to undertake during calibration cannot adversely affect the predictions.
It is now demonstrated that the data driven scheme is conservative for predictions that have this special structure. PROPOSITION 4 (Performance of Data Driven Scheme). Suppose the matrices Σ x , G, Σ δ , Y, Σ ρ define an optimal scheme with posterior denoted by p(p|d) = N (p; µ p|d , Σ p|d ) for arbitrary dataset d.
Furthermore, let C denote a suboptimal simplification matrix and let the matrices F, Σ d v ,G = FGC, Σ δ = FΣ δ F ,Ỹ = YC, and Σ ρ denote a data driven scheme, wherê Σ d v is as specified in Definition 7. Additionally, letμ d p|d andΣ d p|d denote the mean and covariance of the posterior produced by this scheme for the dataset d. 1. If the filtering matrix F is selected such thatG = FG has full row rank, then the estimator matrix E d of the data driven scheme is equivalent to the pseudoinverse of G and can be expressed in terms of the compact SVD In addition, the posterior mean and covariance can be rewritten aŝ IfG = FG has full row rank, and the high fidelity prediction matrix has the form then the data driven scheme is conservative with respect to the optimal scheme.
See Appendix E for a proof. Before providing additional intuition as to the importance of this result, and the role of the filtering term F, it is first demonstrated that the data driven scheme is a generalization of the truncated singular value decomposition calibration method.

Truncated SVD Inversion.
The truncated singular value decomposition inversion or calibration method is a commonly used technique [2,32,51] that generates the solution to the parameter estimation problem using a truncated decomposition of the data matrix. It will be demonstrated that the truncated SVD scheme can be replicated with the appropriate selection of a filtering matrix. To start, consider the SVD of the simplified data matrix Now, letŨ t denote the columns ofŨ that correspond to the largest k nonzero singular values. Furthermore, consider the filtering matrix defined by With this definition, the filtered data matrixG = F TSVDG becomes whereṼ t ,S t similarly denote truncated versions ofṼ,S. Now, as F TSVDG has full row rank (only nonzero singular values are included), the results of Proposition 4(1) allow the estimator matrix for the scheme to be given by the pseudoinverse ofG = F TSVDG and thus can be written as t . The estimated parameter vector of the simple model is now related to the data vector via the inverse of the truncated data matrixG, i.e.
This demonstrates that the truncated SVD inversion method can be replicated with the selection of an appropriate filtering matrix.
It is important to note however that this does not mean that the predictions generated by the truncated SVD method will be conservative. To guarantee this, the prediction must have the form specified in condition (49) of Proposition 4, which in this case requires the prediction matrix to have the form whereṼt contains the columns ofṼ that were removed by the truncation. If this is not obeyed, the predictions may be over confident. Furthermore, this condition is implicitly dependent on the truncation point k, such that this may hold for some values and not others. This dependency explains in part the results of the simulation studies performed by [51] which demonstrated the difficulty in choosing an appropriate truncation point such that predictive uncertainty is accurately estimated by a simplified model.

Selection of Data Filtering.
A key element of the defined data driven prediction scheme is the filtering matrix F. For the results of Proposition 4 to hold, and the scheme to be conservative, two main requirements must be met: 1. F must be selected such thatG = FG has linearly independent rows. This is a fairly trivial requirement, and simply requires that dependent, e.g. duplicated, measurements be combined, e.g. by averaging. Note that the information content is preserved as the error covariance is also transformed. In addition, it requires measurements that are insensitive to the model parameters be dropped. 2. F must be chosen such that the induced separation of the parameter vector v into those components that are estimatable v 1 = V 1 v and lie in the rowspace of FG from those that are not v 2 = V 2 v and lie in the nullspace of FG, forces all components of the parameter vector that have a direct influence on the prediction to be contained in the vector v 2 and are thus not updated during the calibration process. This ensures that these parameters do not take on any surrogate roles, and that the predictions are not corrupted. In addition to these two requirements, the filtering matrix can be selected to improve the predictive performance by removing data components that are only weakly informative to the parameters. This can be considered identical to selecting an appropriate truncation point on a SVD calibration scheme such that the highly informative prior knowledge can be used instead of data that only imposes weak constraints, e.g. as recommended by [32].
However before searching for an optimal filter, it is important to note that for a given type of data, predictions, and simplification, defined by the triple G, Y, C, there may not be an F that allows the prediction to be cast in the form required by Proposition 4 and the use of the data driven scheme is not guaranteed to be conservative. This reinforces the fact that choosing a model simplification and performing calibration with a particular dataset, must be considered an integrated task that is explicitly dependent on the types of predictions that are required.
6.2.5. Summary. This section has introduced a calibration and prediction scheme that is a generalization of the typically used truncated SVD scheme. Furthermore it has been demonstrated that the scheme is guaranteed to produce a conservative prediction posterior when the unmodelled complexity is hidden behind the data. The general structure of the prediction problem, that obeys this condition, is depicted in graphically in Figure 8. This represents the probability densities that define the data driven scheme and includes the dependency structure of the predictions (as required by condition (49) and depicted in Figure 7) as well as the structure of the inflated prior covariance matrixΣ d v (as specified in (46)). The important contribution here is the derived structural condition between the data, predictions and the simplifications that determines when it is adequate to use the data driven scheme, with a simplified model, to assess predictive uncertainty. It is noted that unlike the optimally compensated scheme, this scheme can be used without needing the explicit numerical values within the data and prediction matrices of the high fidelity reference model. It is noted however that, structural information from them is still required to ensure Proposition 4 is met. Although this still may not be straightforward in practice, structural considerations are generally significantly easier to handle than numerical ones.
7. Discussion and Conclusions. This paper has considered what constitutes a simplified, but useful, model. In particular it has examined how simplified models can be used to combine data and expert knowledge within a calibration or inversion process to generate a prediction with a conservative estimate of uncertainty. The concept of an optimal simplified model was defined that determines when standard probabilistic calibration methods are adequate to quantify predictive uncertainty. The main contribution is the introduction of two new calibration and prediction schemes, along with conditions that explicitly define when they are appropriate to generate a conservative estimate of uncertainty, for suboptimal models. These conditions explicitly relate the nature of the calibration data, the predictions of interest as well as the simplifications within the model.
The first scheme allows the optimal posterior distribution to be generated by allowing the simplifications to be overcome by adjusting how the beliefs of the modeller are used to define a prior term. The scheme is only applicable for highly parameterized models. Furthermore, it requires the prior covariance for the parameters of the simple model to be generated using the data and prediction matrices of a high fidelity reference model. This is a significant limitation as if these matrices could be generated for a practical problem they could be directly used to produce a prediction posterior without the need to consider a simplified model. Nevertheless, the value of this scheme lies in defining the ideal calibration process for a simplified model and demonstrates it is possible to overcome suboptimal simplifications through the judicial selection of a prior regularization matrix.
The second data driven scheme is designed for predictions that are strongly related to the data, such that the unmodelled complexity effects both in the same way. This scheme does not require the data and prediction matrices of the high fidelity model to be available for model calibration in the way the optimally compensated scheme does. However, it is only guaranteed to be conservative if the predictions, data and simplification have the required structural form. The key insight provided by this scheme is understanding how the model simplifications can be overcome with the use of the right calibration data.
It was also demonstrated that this data driven scheme is a generalization of the popular truncated singular value decomposition inversion scheme [2]. The generalization allows greater flexibility in filtering the data to ensure that the predictions are of the appropriate form for a given simplified model. Both calibration schemes have been applied to a prototypical groundwater prediction problem in Appendix A.
Finally, it is noted that each of the two newly defined calibration and prediction schemes have conditions that are linked to the data and prediction matrices of a high fidelity reference model. For any practical problem it is unlikely that these conditions can be directly assessed and further subjective judgment will be needed. To make this process easier, two areas of further work can be pursued. Firstly, more synthetic experimental analyses are required to demonstrate how the two schemes can be applied in more complex problems, e.g. as in [51]. The second area is to understand the effect of partial non-satisfaction of these conditions, and determine how the schemes can be made robust to these.
It is also noted that the use of Bayesian networks, for instance as depicted in Figure 10, may be of great benefit to determine when the required structural conditions are likely to be obeyed, for instance, when is a given simplified model optimal, or when is it appropriate to use the data driven scheme. Such structural considerations may also help frame the arguments put forward in modelling projects (e.g. performed within environmental impact assessment studies) as to why a given modelling and calibration approach is adequate for a given prediction problem. For instance these argument can be framed using a two stage process. The first may conceptually link the complex system to an optimally simplified model, this stage would consider the features and characteristics of the system that ideally should be modelled. The second stage may then put forward arguments as to how any further simplifications used to produce a suboptimal numerical model will be handled by the calibration process.
Lastly, several other important areas of further research are identified: • Nonlinear simulators. The analysis within this paper has required a linear relationship between the system properties and the data and predictions. This has allowed generic insights to be obtained, but is a significant limitation and further work is needed to relax this. One approach is to consider higher order expansions of the models such that some of the nonlinearities can be included, for example second order expansions have been considered in [8,31,12]. Alternatively, more direct probabilistic formulations may also be possible that e.g. generalize the data driven scheme and only exploit the structural constraints with the problem. • Nonlinear parameterizations. In the developed analytical framework only linear relationships between the parameters of the high fidelity model and the simplified model where considered. This should be extended to considered nonlinear parameterizations. • Over constrained calibration problems. The results obtained for the optimally compensated prediction scheme required that the simplified model is under constrained by the data and provides no insight to aid the calibration of over constrained problems. Nevertheless, other approaches may be possible that reproduce the optimal, or at least a conservative, result. Similar modifications may also be possible for the data driven scheme. • Conservativeness. This paper has employed a definition of conservativeness based on the mean and covariance of the model predictions. Generalizations of this to non-Gaussian distributions have been considered in e.g. [3,1]. However, it is perhaps more appropriate to consider a generalization of conservativeness that explicitly includes the subsequent decision problem (i.e. engineering design or environmental management). This can be performed using a decision theoretic approach [5] that includes the utility function of this subsequent decision problem and defines an approximate densityp(p) as conservative if the expected utility is not over estimated, e.g.
Here U (a, p) is the utility function that encodes the gain under action a, when the consequence p occurs. Such a generalization would also allow the incorporation of the decision problem into how modelling and calibration should be performed. It is noted that the above generalization is equivalent to Definition 4 when U (a, p) has the form of a weighted squared difference, i.e. U (a, p) = −(a − p) A(a − p) for arbitrary positive semidefinite matrix A.
Furthermore, the uncertainty in the system properties is fully captured by the mean and covariance matrix that defines the Gaussian distribution p(x) = N (x; µ x , Σ x ).
To define the data and prediction likelihood functions, consider the vector x as known and define the mean with the following nonlinear functions, corresponding to the finite difference solution to Darcy's equations [4] G Note the variables , b and q are all known constants. Furthermore, as the numerical simulation at this discretization is believed to be adequate, the data error δ is completely captured by errors in the measurement process such that Σ δ = (0.1) 2 . In addition the prediction error variance is taken to be zero, Σ ρ = 0.
It is noted that the prediction equation has been rewritten on line (59) to explicitly include the data equation as an additive term. This form will be important when judging whether the data driven scheme is appropriate for a particular system simplification.
A.2. Data Generation. The measured head at the observation well is d = 2.5m. This is generated by the data equations d = G(x t ) with no added measurement error. In addition the system properties x t are the same as the prior mean µ x , with the exception that the boundary head is set to h 0 = 1.5m. This difference corresponds to 2/3 of the prior standard deviation.
A.3. Linearized Solution. From the nonlinear functions G(x) and Y(x), a pair of linear functions are constructed by linearizing about the prior mean with a first order Taylor expansion [30,10].
and similarly for Y. Furthermore, a transform into zero mean increments is performed such that ∆x = x − µ x , ∆d = d − G(µ x ), and ∆p = p − Y(µ x ). Under the linearized approximation the data and prediction equations become where the data and prediction matrices are defined by the Jacobian matrices evaluated at the The solutions to the full non-linear prediction problem and the approximate linearized version defined above are given in Figure 11(a). The non-Gaussian posterior density of the nonlinear problem has been produced using a standard Metropolis MCMC algorithm [9] to generate samples that are then smoothed using a kernel density estimator [41]. Overlaid with this is the Gaussian posterior produced from the linearized problem. It is noted that the posterior under the non-linear equations is slightly more peaked and non-symmetric when compared to the posterior generated by the linearized scheme.
It is noted that the non-linearity is only present as the log conductivity is considered to be normally distributed. If, for instance, the hydraulic resistance (inverse conductivity) was represented directly, the equations would become linear. However, this is not pursued here and the error caused by assuming a linear approximation of the nonlinear equations is considered out of scope.
A.4. Assumed Simplifications. Consider the following simplifying assumptions A1: The constant flow boundary condition is assumed known and set to the prior value of 1.0m. A2: Cells 1-5 are grouped together and assumed a homogeneous unit. This set of cells will be referred to as Zone A. A3: Cells 6-10 are grouped together and assumed a homogeneous unit. This set of cells will be referred to as Zone B. This enables the parameters of the simple model to be defined by the vector of log conductivities of the two zones v = [log 10 K A , log 10 K B ] This simplification scheme corresponds to a C matrix of The use of increments allows the state of the high fidelity model to be defined in terms of an increment ∆v such that ∆x = C∆v.
Furthermore, the simplified data equation becomes and similarly for the prediction equation. Now, the linearized form of the simplified data and prediction equations become Here the simplified data and prediction matrices can be written in terms of the high fidelity modelG = GC andỸ = YC, where G and Y are the Jacobian matrices of the high fidelity It is now of interest to determine whether this simplification is optimal. If it is, the naive prediction scheme when used with the simplified model will produce the same results as the optimal scheme applied to the high fidelity model. For the simplification to be optimal, the unmodelled complexity should have no effect on the data or predictions. This occurs when the following conditions hold GUC = 0 and YUC = 0 where the columns of UC define an orthonormal basis for the cokernel of C, as defined in (17). The unmodelled complexity consists of h 0 and the two 4D parameters that describe the small scale heterogeneity in the conductivity of the two zones. It is noted that this condition does not hold and thus the simplification is not optimal. Furthermore, the failure of this condition to hold is completely due to assumption A1 that considers the boundary condition h 0 is known. If this assumption is removed and h 0 included in the simplified model, while at the same time retaining the homogeneity assumptions of A2 and A3, then the simplification would become optimal (under the linearization considered). This optimal simplification will not be considered and it will be expected that the naive method will not be conservative.
Finally, the structure of the high fidelity prediction problem and the simplified problem is depicted graphically with two Bayesian networks in Figure 10. It is noted that the suboptimality of the simplification can be observed in Figure 10(b) as the data and predictions are not conditionally independent of the unmodelled components (h 0 ) given the model parameters (K A and K B ).
A.5. Prediction Results. Here, the performance of the two new calibration and prediction schemes are considered, with the resulting posterior density functions for the predicted head displayed in Figure 11(b). The results are also compared with the naive scheme which represents a typical probabilistic calibration scheme of a simplified model. It is important to note that no "ground truth" value for the prediction is given for comparison purposes as the objective is to produce the full probability distribution and not make a single point prediction.
A.5.1. Naive Scheme. The naive scheme considers the uncertainty in the parameters included by the simple model, but ignores any errors introduced in the simplification. As the simplification is suboptimal, it should not be expected that the prediction posterior will be conservative. Furthermore, the results in Figure 11(b) demonstrate that this is true for this scenario as the posterior is overly narrow.
To understand the cause of the overconfidence in this scenario, consider the prior covariance in the parameters ∆v as defined in Proposition 1 (60) Σ v = C † Σ x C † ≈ 10 −2 8.58 6.19 6.19 8.58 Due to the large range of 300m used for the variogram of the log conductivity field, the correlation between the two parameters maintained by the simple model is considerable. In addition, the known flow rate and the assumption of known boundary head h 0 , allows the head measurement to be used to very accurately, but incorrectly, estimate the conductivity K A . Furthermore, due to the large correlation that K A has with K B , this scheme produces an overconfident prediction probability density function for the parameter K B , which in turn causes the predicted head to be also overconfident and non-conservative.
A.5.2. Optimally Compensated Scheme. Now consider the optimally compensated calibration and prediction scheme. Firstly it is noted that the simplified problem is highly parameterized as D v ≥ D p + D d as D v = 2 and D p = D d = 1. The major difference between the naive scheme and the optimally compensated scheme is the modification of the prior covariance used for the parameters of the simple model. In particular, the modified prior of this scheme is not just defined in terms of the simplification matrix, but also in terms of the data and prediction matrices, via the matrix R =Z † Z, where Z = G Y andZ = G Y . The optimally compensating prior covariance becomeŝ It is noted that the only difference between the values of the naive prior covariance Σ v and Σ o v , is the inflation of the marginal variance of the first component corresponding to log K A . This causes less information to be propagated from the data into log K B , and enables the prediction posterior to replicate the results of the high fidelity model.
The key difficulty with this scheme is that it requires the data and prediction matrices of the high fidelity model in order to produce the inflated prior covariance matrixΣ o v that compensates for the suboptimal simplifications.
A.5.3. Data Driven Scheme. To apply the data driven calibration and scheme, it is first of interest to determine if it is appropriate for the problem, in particular, does the prediction have the right structural relationship with the data and the simplifications as specified in Proposition 4. This will be checked first using the algebraic expression, and then using the structural characteristics of the network depicted in Figure 10 The required algebraic condition requires that for some A and B the prediction matrix can be written as For this problem, the data is scalar, and thus the filtering matrix will be set to unity F = 1. Also, from the prediction equation in (59), it is clear that the prediction matrix Y is equal to the data matrix G plus an additional term, which will be denoted by T. This allows the Jacobian matrix Y to be written as Thus, the matrix A in (61) will be set to unity A = 1. Now it is of interest to consider the second term in (61). For the prediction to have the required form, the matrix T = Y − G must be equivalent to BV 2 C † , and thus must satisfy two conditions: The first condition (63) requires the difference between the data and the prediction to be independent of the unmodelled complexity. For this scenario this can be checked algebraically using the high fidelity prediction and data matrices and is satisfied. Now consider the second condition (64), this requires the difference between the data and the prediction to be independent on the rowspace components of theG. This can be easily checked using just the simplified data and prediction matrices and is also satisfied. Thus, it is guaranteed that the data driven scheme is conservative for this problem.
As an alternative to this algebraic check, the conditional independence requirements can be check using the structure of the Bayesian network. In particular, the requirement of (61) is equivalent to the requirement that the prediction p = h 10 is conditionally independent of all unmodelled components, u = [h 0 , K − A , K − B ], given the uncorrupted data g = h 5 and the nullspace components of the parameter vector v 2 = K B . From the structure of the network, these two nodes are the only parents of the prediction and thus, the required independence requirement is satisfied. This graphical approach provides much greater intuition into when the requirements are met.
When the data driven scheme is applied, it is noted that the available prior knowledge is modified in two ways. Firstly, the prior information on K A is ignored and this parameter is estimated from the data only. Secondly, the correlation between K A and K B is also ignored, this causes the posterior over the parameter K B to be the same as the prior. These modifications are encapsulated in the data driven prior covariance matrix The posterior generated by this scheme is conservative and is depicted in Figure 11(b). It is noted that the posterior is slightly shifted than that produced by the optimal scheme and has a larger variance.
A.6. Summary. This section has demonstrated the operation of the naive, optimally compensated, and data driven calibration and prediction schemes on a prototypical groundwater problem. A high fidelity model was considered to represent the true belief of an modeller and a suboptimal simplification developed to represent a computational model.
The key difference between the three schemes is in the specification of the prior covariance matrix for the parameters of the simplified model. In particular, the naive scheme generates this by directly projecting the prior distribution that exists in the parameters of the high fidelity model into the parameters of the simplified model. The optimally compensated scheme performs the projection using full knowledge of the data and prediction matrices of the high fidelity model. Lastly, the data driven scheme uses the same matrix as the naive scheme but throws away some of the information it contains.
The conditions under which the schemes are conservative (or optimal) have also been highlighted. The naive scheme should only be applied to optimally simplified models, this was not satisfied and the posterior shown to be overconfident and non-conservative. The optimally compensated scheme should only be applied to highly parameterized models. And the data driven scheme is conservative only when the predictions and data have a similar dependency to the unmodelled complexity. These last two conditions were shown to hold for the problem considered, and the generated posterior distributions were also conservative.
Appendix B. Proof of Proposition 1.
Proof. For a given simplification matrix, the parameters of the high fidelity model x can be expanded using (18) as Furthermore, from the decomposition in (17), the matrix C UC can be rewritten as This is nonsingular and has an inverse that simplifies to where C † = V C S −1 C U C . Thus, the transformed covariance matrix in the space of v, u has the form Appendix C. Proof of Proposition 2.
Proof. Under the conditions of part (1) the matrix C denotes an optimal simplification and thus GUC = YUC = 0, where UC is an orthonormal basis for the cokernel of C, e.g. as given in (17). Using the expansion of the parameter vector as x = Cv + UCu, the covariance Σ x becomes Substituting (65) into the posterior mean and covariance of the optimal scheme, defined in (10) and (11), and noting that GUC = YUC = 0, the following forms are obtained Noting thatG = GC andỸ = YC these expressions are equivalent to mean and covariance those of the naive scheme, i.e.μ n p|d = µ p|d andΣ n p|d = Σ p|d for all d. This proves part (1) of the proposition.
To proceed, to parts (2) and (3), for the naive scheme to be conservative, the mean and covariance of the posterior must obey (30), which simplifies to (66)Σ p|d E p(d,p) (μ n p|d − p)(μ n p|d − p) Also, the expectation over d, p is equivalent to an expectation over the independent variables x, δ, ρ, where d = Gx + δ and p = Yx + ρ. Thus to prove part (2) of the proposition, it will be now shown that this holds under the special condition of YUC =ỸE n GUC, defined in (32).
To start, note thatμ n p|d =ỸE n d, d = [Gx+δ] and x = Cv+UCu, thus the differencê µ n p|d − p can be written Where, (68) has used condition (32). Now, as v = C † x, δ, ρ are all independent and zero mean, the expected squared difference ofμ n p|d − p can be written in terms of the covariance matrices Expanding the right hand side with the naive estimator matrix E n = Σ vG (GΣ vG + Σ δ ) −1 and simplifying, produces the result E (μ n p|d − p)(μ n p|d − p) Thus, the special condition YUC =ỸE n GUC is sufficient for (66) to be satisfied (with equality), ensuring the naive scheme is conservative. This proves part (2). To prove part (3), consider equation (67) above and assume v and u are independent and YE n GUC = YUC, then the expected squared difference ofμ n p|d − p becomes Defining M =ỸE n GUC − YUC = 0, this simplifies to E (μ n p|d − p)(μ n p|d − p) =Σ n p|d + MΣ u M . Now, as M is a non-zero matrix and Σ u is a non-zero positive semi-definite covariance matrix, MΣ u M must be non-zero and positive semi-definite. Thus, the required condition (66) does not hold as E (μ n p|d − p)(μ n p|d − p) Σ n p|d . and the scheme is strictly not conservative. This proves part (3).
This can be used to directly define the mean of the prediction posterior With similar substitutions as above, and using the Woodbury identity [42, 15.3(b)(i)], the covariance of the prediction posterior simplifies tô where W = V 2 V 2 = I − V 1 V 1 represents a projection on to the nullspace ofG . This concludes the proof of part (1) of the proposition.
For the scheme to be conservative, the mean and covariance of the posterior must obey Also, the expectation over d, p is equivalent to an expectation over the independent variables x, δ, ρ, where d = Gx + δ and p = Yx + ρ. Now, the difference between the prediction p = Yx + ρ and the meanμ d p|d simplifies to Furthermore, as v = C † x, δ and ρ are all independent and zero mean, then the difference p −μ d p|d has an expected value of zero and a covariance given which is equivalent toΣ d Thus, the required condition (72) is satisfied (with equality) and the prediction scheme is conservative. This proves part (2) of the proposition.