{'fraction_non_alphanumeric': 0.02553230712508065, 'fraction_numerical': 0.006083510000921744, 'mean_word_length': 5.032527105921601, 'pattern_counts': {'":': 2, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A method to reduce model uncertainty by fusing the output from multiple stochastic simulators

In virtualizing engineered systems, it is essential to come up with simulators that are essentially capable of representing the system in its “as-deployed” state. Any attempt to this end may only be approximate given the inherent uncertainties present in the loadings and operational conditions of the system, as well as the configuration of the system itself (geometry, materials, control systems, boundary conditions, etc.). This is especially true for complex systems, such as wind turbines, where often a number of assumptions govern the setup of the engineering models. Such models are often made available at different granularities with each one offering a diversified level of precision depending on the quantity of interest (e.g. macroscopic displacements or microscopic strains) and the properties of the acting loads (e.g. amplitude and frequency content). This implies that the predictive capabilities are severely hampered when a single so-deemed best model is chosen for simulation. Building on this idea, we here present a method for fusing the outputs from multiple simulators (e.g. aero-servo-hydro-elastic simulators) for estimating a quantity of interest (QoI) with higher precision. The proposed ensemble learning approach comprises two main building blocks. Firstly, a clustering step by means of a Variational Bayesian Gaussian mixture model, employed for the weighing of each available simulator. Clustering is performed on the basis of the binned input space, which allows for extraction of a probability map for each local region of the binned input space. This delivers an adaptive scheme, which allows different simulators to more or less prominently contribute to the prediction of the QoI, depending on the range of the input parameters. Local weighted Bootstrap Aggregation is then executed in a second step for combining the clustered ensemble of outputs from the individual simulators. A simulated toy example and a wind turbine blade fatigue case study are herein exploited to demonstrate the efficacy of the suggested ensemble learning scheme. The approach is compared against alternatives typically adopted in existing literature, such as Stacking, classical Bagging, and Bayesian Model Averaging. The results confirm an improvement in predictive capabilities as expressed via the reduction in the generalization error and the narrowing of the associated confidence interval.


Introduction
The predictions from stochastic numerical simulators used in the design of machines and structures are subject to uncertainties. There are several sources which contribute to these uncertainties: physical (input), parametric, statistical, measurement and model-form uncertainties. Model-form uncertainty refers to the uncertainty in the mathematical and computational structure of the simulator itself and the assumptions behind it. The uncertainty due to model-form, is not usually quantified and sometimes it is not even acknowledged. Some reasons for this are the absence of consensus on how to quantify and represent model  2 uncertainty, and lack of data to do proper benchmarking [1]. However, when making critical decisions concerning the design and safety of the mechanical and structural components of wind turbines, there is a great need to deal with model uncertainty. If the model uncertainty is not taken into account, the resulting decision may be erroneous or sub-optimal. In many instances, stochastic predictions from multiple simulators for the same quantity of interest are available. In this paper, the term "simulator" is used to encompass any numerical computer model simulating a physical phenomena. In the wind energy domain, aero-servohydro-elastic simulator-to-simulator comparisons have been performed with relevant results presented in reports and publications, for instance [2,3]. Often, the simulators deployed in these comparison and benchmarking activities have been calibrated and validated via experimentation, and have been subsequently adopted by manufacturers in the design and deployment of wind turbines. The problem arises, however, when one is unable to definitively prove that a single simulator is ubiquitously superior to the others across the full range of acting environmental and operational conditions and for the various materials and components making up the various sub-structures of the wind turbines. We contextualize the problem by offering an example where the stochastic blade root bending moments of a wind turbine are predicted via several distinct aero-servo-elastic simulators, which implement multiple solution methodologies to solve the same problem. For instance, the structural dynamics in one simulator may implement a multi-body/Finite Element formulation, and another simulator may implement a classical modal-superposition approach. The aerodynamics in one simulator may implement a Blade Element Moment theory, whereas the actuator line based computational fluid dynamics (CFD) or a blade resolved CFD formulation may be adopted in another [4]. Hence, whether one is performing reliability-based design optimizations or calibrating safety factors, or calculating design loads, etc., having the means to evaluate the model-form uncertainty and possibly decreasing it becomes crucial. One popular approach in the wind energy field is based on the multi-fidelity sampling or surrogate modelling approaches [5,6,7]. Another approach approach lies in fusion of the predicted outputs from alternate simulators, thus exploiting their individual predictive traits, as opposed to seeking a single optimal-fit option [8]. The idea is akin to model averaging; an approach often adopted within the context of statistical inference. However, typical multi-fidelity or averaging or aggregation schemes, as reported in existing literature, do not make distinction on the influencing region of the input space, or might further assume one of the models in the ensemble as reference, i.e., the data-generating model. Realizing that the range of inputs is a salient parameter to what concerns modeling of wind turbine systems, we here present an unsupervised ensemble learning method, which treats individual simulator outputs as components contributing to an ensemble estimate, as opposed to competing predictions. The proposed scheme succeeds in addressing both the epistemic uncertainty owed to lack of training data, as well as the inherently present model form uncertainty. We call our method, originally introduced in recent work of the authors [9], Unsupervised Local Cluster-Weighted Bootstrap Aggregation (UnLoCWeB).

Method
The method exploits the probabilistic output form a diversified set of individual simulators (engineering models of different granularity). It proceeds to firstly extract the local weights of these simulators via an unsupervised variational Bayesian Gaussian mixture (VBGM) scheme, followed by estimation of a weighted ensemble, delivered by Bootstrap Aggregation (or weighted Bagging).
We initiate with discretization of the input space domain into a finite number of bins (∆x). Each bin, or rather the data contained therein, is further segregated into a training and validation set. The output from the simulators is clustered via use of VBGM to the training where K is the unknown number of clusters, each having a mixing coefficient π k and parameters of Gaussian distributions: mean µ k and covariance Σ k . The posterior distribution of π k , µ k and Σ k are computed using Variational Bayesian inference. An adaptive solution (VBGM) is selected since the number of clusters needs to be adaptively defined, and may not be considered as fixed a priori. Furthermore the application of the clustering procedure in bins of the input space, as opposed to the full range, favors flexibility and allows a possible varying weighing of each simulator, conditioned on the acting range of the influencing inputs. As an added safety, we include an intermediate step, which examines the stability of the cluster on perturbed versions of the data included within each bin.
The probability map (weights) of the simulators' output is then extracted as the joint probability P S i , C j | ∆x of a cluster C j and a simulator {S i }: where P C j | ∆x = N C j |∆x/N, N C j |∆x denotes the number of local data points contained in cluster C j and N corresponds to the number of available output samples (from all Local cluster-weighted Bootstrap Aggregating (Bagging) follows next [10], to fuse the ensemble of outputs from each bin -based on the previously extracted weights -into one single aggregated predictor. The bagging algorithm is comprised of the following steps: by randomly drawing n ∆x times with replacement from the local clustered data, h n (·) defines an estimator as a function of the data, (iii) repeat steps 1 and 2 M times to infer ĝ k * , k = 1, . . . , M . The final local cluster-weighted bagged estimator is obtained as:

Analytical computational experiment
We present a synthetic analytical example to motivate the approach in the context of response prediction for wind energy infrastructure, and demonstrate the benefits and limitations. Assume that a target QoI (blade or tower response) is prescribed by the following analytical expression: We further make an assumption on the QoI being normally distributed: The coefficient of variation COV is assumed to be homoscedastic, while the random variable X is uniformly distributed X ∼ U (0, 1). Five alternative simulators are assumed available for predicting Y target . For the purpose of this illustrative example it is assumed that all simulators may be attributed the same level of confidence, which is often the case in absence of experimental results or field measurements, serving for validation. To further highlight the workings of the proposed scheme, we assume all simulators to be in mutual disagreement and to further deviate from the target response (as illustrated in Figure 1). The form of each simulator is listed below The model coefficients are selected so as to induce the desired mutual disagreement, and are here defined as The set of available simulators suffers different levels of approximation in terms of the overall response behavior (linear versus non-linear), the calculated expected value (systematic versus random bias), and even the position of extrema of the response with respect to the value of the inputs. As is further evident, as the generating model, Y target , is not included in the candidate model set, this example pertains to an M-open setting. The goal lies in predicting the stochastic response conditional on ∆x, upon availability of the individual outputs of the five stochastic simulators: sults are compared against the commonly adopted alternative of classical bagging, in terms of the generalization error (GE), and the breadth of the 95% empirical bootstrap confidence interval (CI). The GE, which comprises a global metric of the estimation precision, is obtained as: where n v denotes the validation set dimension, and µ Y corresponds to the mean of the computed simulator response for the validation set. The local predictive capability is reflected is the computed 95% CI. In order to assess the performance of the proposed scheme, simulations are varied with respect to the number of stochastic samples per simulator and the assigned coefficient of variation (COV). Weighted Bagging is executed for 100 Bootstraps, with the outcome averaged across 50 repetitions, to ensure independence from a specific split of the sample space. Figure 2 indicates improvement of the prediction for the proposed approach, captured in Y, versus classical Bagging. Specifically, reductions of 39% and 7% are obtained for the the generalization error ( Figure 3) and the 95% confidence interval (Figure 4), averaged across the binned input space. A further assessment is offered with respect to the variation of the output COV. Assignment of a higher output dispersion is expected to harden the discrimination of unique clusters. Indeed an increasing output COV, according to  result in deterioration in the average reduction in the GE and CI of our proposed scheme versus a classical Bagging approach. This is particularly true for a COV ≥ 10%. This is tackled by increasing the number of utilized samples to 5000 (per simulator) which results in improvement of the proposed approach versus classical Bagging. This behavior is explained by the fact that when clustering with a mixture of Gaussian distributions, the majority of the data is assumed to lie close to the center of the mixture component. A higher number of samples enables the discrimination of unique clusters, and therefore more unique weights.

Comparison to other methods
For the comparison to other methods, we modify the previous example and use 4 simulators given as: The constants are now set to {A 1 = 1, B 1 = 10,  In further providing a comparison to the state of the art, we contrast the predictions delivered by our proposed UnLoCWeB scheme to (i) Bayesian Model Averaging, and (ii) Stacking of predictive distributions (Stacking). The BMA and Stacking methods are implemented according to the work of [11] using PyMC3 [12]. The simulations are conducted with 500 samples per each of the 4 simulators, a discretized input bin size ∆x = 0.02, and COV = 0.05. The target is once again the expected value of the response under availability of outputs from all individual simulators. The stacking weights are determined equal to 0.88 for simulator 1 and 0.12 for simulator 3. The BMA select a single optimal simulator via assignment equal to 1 for simulator 1. The generalization errors for each method are listed in Table 1. Simulator 1 is in general not an adequate simulator for the target QoI, but performs better than the individual remaining simulators. In the BMA setting this leads to a higher marginal likelihood assignment when compared against the further candidates. This result pertains across the whole input space. The Stacking scheme further assigns significant weight to simulator 3, yielding an improved generalization error of 0.602 (as opposed to 0.723 for the BMA). To the contrary, the adaptive scheme proposed in the UnLoCWeB approach, achieved by the local assignment of weights with respect to the input space, delivers a lower generalization error of 0.511, and thereby a higher precision amounting to a 15% reduction, as compared against Stacking.  Table 2. In terms of acting loads, the turbulent wind field ( Figure 5) is obtained via use of an exponential coherence model and the Kaimal turbulence auto-spectrum following Annex B in [13]. A quasi-static Blade Element Momentum model is used to calculate the normal F N and tangential F T aerodynamic forces along the span of the blade (Figure 6). For the structural modeling, a common engineering approximation is adopted. The blade is approximated as an equivalent clamped-free tapered beam, which represents the main load bearing spar caps and shear-web configuration along the blade span. The target QoI in this case is the time varying blade root in-plane bending moment M X , which is induced as a result of the action of the nodal forces F N and F T along the span of the blade. Ten finite element simulators are used as candidate models for reproducing this QoI. The configurational characteristics of the FEM simulators are overviewed in Table 2, with a further illustration for three characteristic such simulators included in Figure 7. 500 times series of F N and F T , are generated as inputs. Each time series is of 600 seconds in duration, at a time step of 0.01 sec, and the forces are assumed to be exerted onto the cross-sectional aerodynamic center.
where n i designates the number of load cycles with range ∆M X,i , i denotes the fatigue cycle index, and N eq specifies the equivalent number of load cycles, typically 10 7 cycles. A Whöler exponent m = 10 is assumed, as typical for a composite structure.   **"In-plane projection": the resultant of the forces is projected onto the YZ − plane. **"According to γ": the forces are oriented according to the angle γ, which depends on the relative wind speed, the blade pitch and twist angles.  QoI, i.e., the damage equivalent fatigue of the blade root bending moment. For low amplitudes, the linear nature of the response results in the simulators exhibiting similar predictions. This observation justifies the nature of our proposed adaptive calculation scheme, which employs binning to isolate regions of effective discrepancy between candidate simulators, ignoring a priori information on their nominal precision. According to Figure 8, the UnLoCWeB predictor produces an ensemble estimate, which is characterized by a lower variance, with respect to the dispersion exhibited by the individual simulators. This indicates lowering of the uncertainty in the estimation of the short-term fatigue DEL. The clustering step effectively delivers a map of probabilities, in the form of the joint probability P S i , C j | ∆x of a cluster C j and a simulator S i . The exploitation of these probabilities in the bootstrapping step results in a shift of the ensemble predictor towards those clusters which comprise a higher density. This observation is verified when observing the estimation result at wind speeds above 15 m/s, where simulators 5 and 6 produce drifting estimates, in comparison to the main cluster.
In a final investigation, we explore the effect of removal of the highest fidelity simulator 10 from the candidate set. As reported in Figure 9, the UnLoCWeB estimation is hardly affected. This implies that the proposed scheme ensures a robust estimation, which is maintained even in absence of the higher precision (and most computationally taxing) simulator. The computational time for simulator 10 is approximately one order of magnitude higher than the simpler simulators. In this respect, we remind the reader that this trait is conditional to the examined QoI. A reliability or failure analysis focusing on extreme responses, or non-linear effects could instead require an alternate weighting of the higher precision simulator.

Concluding Discussion
We proposed an ensemble learning framework, termed UnLoCWeB, which relies on unsupervised variational Bayesian Gaussian mixture clustering and local weighted bootstrap aggregation of outputs from individual simulators.
• Clustering is used to infer a probability map, serving for weighing of the simulators. • An adaptive scheme is proposed with clustering executed with respect to bins of the input space. • In contrast to existing methods, this scheme assigns weights to clusters of the predicted output, within each bin of the input space, not to the individual simulators. • A cluster-weighting bootstrap aggregation is then carried out for each bin of the input space for rendering an ensemble prediction.  Figure 8: Comparing the DEL of the blade root bending from 10 FEM simulators, and the local cluster-weighted Bagged predictor with the 95% CI. The framework is here illustrated in the context of predictions for engineering systems where response variability is noted across the range of operational loads, and where the assumed properties of the engineering models can influence response predictions. Wind turbine structures comprise such challenging systems due to the high loads these are exposed to, and the numerous simulators at hand (even in standard practice) for simulating these systems. Both toy examples, and a more applied case study is offered. In both cases an improvement in performance and an overall reduction in the generalization error and the 95% confidence interval are noted.
The performance of the proposed UnLoCWeB scheme is contrasted against available alternatives, namely Bayesian Model Averaging (BMA) and Stacking of predictive distributions. It is observed that BMA tends to favor a single simulator, across the whole input space. Stacking results in assigning further weight to a second additional simulator, which improves prediction with respect to the BMA, yet proves sub-par when compared against the UnLoCWeB scheme. The latter succeeds in reducing the generalization error by an additional 15% compared to Stacking. This is particularly promising for applications where available simulators tend to under-perform in different regions of the input space.
A challenging issue with the proposed method is when the correct simulator output is clustered separately (as an outlier) from the larger density clusters of the remaining simulators for certain regions of the input space. In other words, the derived weights may be inadequate because the method fails to recognize that certain simulators are more fitting than others in certain regions of the input space. We envisage three possibilities to tackle this problem: (1) include strong priors on the weights of the individual simulators. (2) verify if the sodeemed correct simulator is a systematic outlier across all regions of the input space and across several QoI. (3) a possible insight is that measurements (when they become available) may be considered as yet another physical simulator(s) in the mix with other numerical simulators, with the proposed algorithm simply re-run without any additional mathematical manipulations. We argue that such a way of performing data assimilation is critical with measurements (preferably with more than one replication), in order to update ambiguous clustering weights.
Finally, follow-up work will explore the case where the QoI follows an extreme or skewed