{'fraction_non_alphanumeric': 0.03309833910861513, 'fraction_numerical': 0.013950292747042658, 'mean_word_length': 4.684666327050433, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Robustness of model averaging methods for the violation of standard linear regression assumptions

In a regression analysis, a single best model is usually selected among several candidate models. However, it is often useful to combine several candidate models to achieve better performance, especially, in the prediction viewpoint. Model combining methods such as stacking and Bayesian model averaging (BMA) have been suggested from the perspective of averaging candidate models. When the candidate models include a true model, it is expected that BMA generally gives better performance than stacking. On the other hand, when candidate models do not include the true model, it is known that stacking outperforms BMA. Since stacking and BMA approaches have different properties, it is difficult to determine which method is more appropriate under other situations. In particular, it is not easy to find research papers that compare stacking and BMA when regression model assumptions are violated. Therefore, in the paper, we compare the performance among model averaging methods as well as a single best model in the linear regression analysis when standard linear regression assumptions are violated. Simulations were conducted to compare model averaging methods with the linear regression when data include outliers and data do not include them. We also compared them when data include errors from a non-normal distribution. The model averaging methods were applied to the water pollution data, which have a strong multicollinearity among variables. Simulation studies showed that the stacking method tends to give better performance than BMA or standard linear regression analysis (including the stepwise selection method) in the sense of risks (see (3.1)) or prediction error (see (3.2)) when typical linear regression assumptions are violated.


Introduction
When we conduct a regression analysis, it is common to choose one model which best explains the response variable. On the other hand, there exist the cases when the response variable can be better predicted by combining several plausible models. Both Staking and Bayesian model averaging (hereafter, BMA) (Madigan and Raftery, 1994;Wolpert, 1992) use a concept of combining candidate models with a carefully chosen candidate model weights. In the stacking, weights are chosen to minimize the sum of the squares of the distances between response variable values and a linear combination of predicted values obtained by each candidate model. On the other hand, BMA takes advantage of the posterior probability to combine candidate models, and the weights are selected by the relative contribution of each model compared to other candidate models using approximated Bayesian information criterion (BIC).
Even though the stacking and BMA share the similarity in the viewpoint of fusing candidate models to get a better model using model combining weights, each method is based on different types of combining weights and, consequently, they may show dissimilar results in prediction or accuracy of parameter-estimates under various circumstances. When candidate models include a true model, BMA gives a better performance than stacking or, at least, provides performance as much as stacking in perspective of the distance between the regression coefficient parameters and its estimates ( β−β 2 , the risk) (Clarke, 2003). In contrast, when candidate models do not include a true model, it is known that the stacking outperforms BMA in prediction or estimating coefficients (Clarke, 2003).
The idea of combining several candidate models can be considered as "ensemble learnings" in general, and many ensemble learning methods have been suggested under various frameworks. In order to improve performance in classification problems, bagging (Breiman, 1996a) and random forest method (Breiman, 2001) were introduced, especially for tree models, and Schapire (2003) proposed boosting, which combines trees based on weighted samples. For the regression problems, stacking and BMA are two representative methods. The stacking was originally proposed by Wolpert (1992) and has been developed by Breiman (1996b). Hjort and Claeskens (2003) suggested a more sophisticated version of the stacking method, say "Frequentist Model Averaging (FMA)", by establishing a largesample likelihood framework to accurately describe properties of model averaging estimators. An optimal weight choice for the FMA has been studied by Liang et al. (2011). For a sparse model averaging, Ando and Li (2014) proposed a methodology of selecting model averaging weights under a high-dimensional regression setting, and it was also extended to the generalized linear models case (Ando et al., 2017). Recently, Zhang et al. (2020) studied asymptotic distributions of weights for the model averaging when the number of parameter diverges. On the other hand, BMA approach has been developed by Madigan and Raftery (1994) and Kass and Raftery (1995), and Hoeting et al. (1999) provided elaborate reviews of BMA applications. A study of priors for BMA can be found in Fernandez et al. (2001), and Eklund and Karlsson (2007) suggested a new approach of BMA using predictive likelihoods. Note that most of studies have been focusing on either suggesting a new framework of selecting weights under different circumstances or studying an (asymptotic) theoretical properties of the estimator of weights. However, since there exist various versions of the model averaging methods, it would be practically useful if we have a guideline of choosing the best model averaging method according to a specific situation.
In the standard regression analysis, in which necessary assumptions should be satisfied, it is often the case that we have a dataset which does not satisfy one (or many) of the assumptions. Detecting such a violation sometimes is not straightforward even with rigorous regression diagnostics. Furthermore, it could be challenging to take an action to rectify such a violation if we do not have enough information about the given dataset or the true model. Therefore, it is worthwhile to investigate a way of providing a robust statistical model even though the standard linear regression assumptions are violated. The purpose of the study is to compare the results of model averaging techniques with traditional regression model when several model assumptions are not met. We conduct several simulation studies under different scenarios. Each scenario is designed to violate one of standard linear regression assumptions, and we compare the model averaging results with one from traditional linear regression model including the model selected by the stepwise method.
The remainder of the paper is organized as follows. In Section 2, we introduce the stacking method and Bayesian model averaging, and discuss how to construct a list of candidate models for the model averaging methods. In Section 3, several simulation studies are conducted to compare performance of the stacking method and BMA as well as the result of the linear regression with full model and a model constructed by the stepwise selection. The simulation deals with two scenarios: (1) fitting a regression model with a dataset containing many outliers, and (2) fitting a regression model when a distributional assumption is violated. Here, we consider a dataset whose error terms come from a heavy-tailed distribution or whose response variable actually follows a Poisson distribution. In Section 4, we compare performance among aforementioned methods using the water pollution dataset. Finally, Section 5 summarizes the findings and discusses a direction of further studies.

Problem formulation
Suppose that we have p predictor variables X = (X 1 , . . . , X p ) to explain a response variable Y, and define X ∈ R n×p and y ∈ R n×1 as corresponding collections of n data points, respectively. Denote x i = (x i1 , . . . , x ip ) ∈ R p×1 be the i th observation of X (or i th row of X), and, similarly, let y i ∈ R be the i th datapoint of Y.
A standard linear regression model is expressed by, and let us consider this as a true model. Letf (x i ) be an estimator of the true model, whereβ k , k = 0, 1, . . . , p, is an estimator of β k based on the datapoints X and y.
The model averaging combines several models instead of selecting a single model, and, consequently, we need to take account of a set of properly selected candidate models. Suppose that we have a list of candidate models, M M M, consisting of M candidate models, M M M = {M 1 , . . . , M M }. In the standard linear regression model, various combinations of predictor variables can be considered as a list of candidate models, that is, M m ⊆ {X 1 , X 2 , . . . , X p }, for m = 1, . . . , M. For example, M 1 = {X 1 , X 2 , X 3 } represents a candidate (linear regression) model using X 1 , X 2 , X 3 only. Given a candidate model, M m , define an estimator of f (x) asf where for m = 1, . . . M and k = 1, . . . , p, otherwise.
β (m) k , k = 1, . . . , p is the ordinary least square (OLS) estimator of the regression coefficient of X k in M m , andβ (m) 0 indicates the OLS estimator of the intercept in M m . Therefore, we can construct a matrix containing all the OLS estimators across M candidate models such that, Note that some of elements of m th column ofΓ Γ Γ are zeros if corresponding predictors are not included for the candidate model M m .

Stacking
Stacking method (or Stacked Generalization, Wolpert (1992)) is a model averaging technique from a frequentist viewpoint (Friedman et al., 2001). When it comes to a standard linear regression model, the stacking provides an improved prediction accuracy by forming a linear combination of different candidate models (Breiman, 1996b). Specifically, in the stacking method, coefficients are chosen to minimize the sum of squared of differences between values of the response variable and a linear combination of predicted values from fitted regression model based on different candidate models.
Since the combining models based on the whole training dataset may cause a problem of overfitting in the stacking, the leave-one-out method has been considered when obtaining predicted values. Letf −i m (x) be a fitted model using predictors in M m based on the training data in which i th observation is eliminated. Thus, a possible way to achieve a stacking weight can be formulated by an optimization programα (2.5) whereα α α = (α 1 ,α 2 , . . . ,α M ) (Friedman et al., 2001). Usingα α α, we can construct a stacking estimator based on a given M candidate models. That is, stacking estimators of coefficients and intercept term in (2.2) can be expressed asβ Various determinations of the combining coefficients can be obtained by putting a constraint on the weights. The five types of restriction are often considered to compare a performance of the stacking method. The first model (S1) does not impose any constraint on the weight, and Clarke (2003) suggests three types of restrictions to improve performance of the stacking method: • Restriction Type 1 (S2): M m=1 α m = 1, • Restriction Type 2 (S3): From S2, by replacing negative α m with zero and then re-normalizing so the nonzero α m 's sum to one. Also, we consider an additional restriction type, which gives competitive performance empirically: • Restriction Type 4 (S5): α m ≥ 0 for all m = 1, . . . , M.
Note that S2 (or S4) is not obtained by normalizing S1 (or S5), because each restriction is considered at the optimization process.

Bayesian model averaging
To construct a parsimonious model, we usually select significant (explanatory) variables based on the p-values. However, such a process may mislead the result by selecting inappropriate variables against the intuition or disregarding model uncertainty (Raftery, 1995). BIC has been suggested as an alternative way for model selection. BMA method based on the BIC suggests a reasonable framework to compensate the model uncertainty.
In the linear regression model (2.1), we can express a posterior probability of β β β := (β 0 , β 1 , . . . , β p ) given the data x as (2.7) In the linear regression model, the BIC approximation plays an important role to approximate P(M m |x) in (2.7) (Raftery, 1995) as follows: where R 2 m is the coefficient of determination for the fitted linear regression model using the predictors in M m , and p m indicates the number of predictors in the model M m .
Under the given candidate models, M 1 , . . . , M M , estimators of regression coefficients can be derived by using their posterior means such that, for k = 1, . . . , p, (2.10) Practically, E[β k |M m , x] in (2.10) under the candidate model M m can be estimated by its maximum likelihood estimator (MLE),β (m) k , and combining it with the approximation from (2.8) giveŝ (2.11)

Selecting candidate models
Selecting a list of candidate models is one of the essential part of the model averaging method because the performance of the method heavily depends on the selected candidate models. For example, if a list consists of implausible models to explain the data, the performance of the averaging method would be even worse than the ordinary regression analysis. However, there exist few of studies to choose ideal candidate models with a solid theoretical support. Breiman (1996b) suggested choosing p candidate models for p dimensional data. Specifically, suppose that we have a list of candidate models M = {M 1 , . . . , M M }. A candidate model M m has m predictors for m = 1, . . . , M (= p). The m predictors can be selected by comparing the R 2 (coefficient of determination) among all the possible linear regression model with m predictors. However, when p gets larger, there exist 2 p possible candidate models to examine, which possibly results in an inefficient process to select candidate models. The leaps-and-bound algorithm (Furnival and Wilson, 1974) can be used to choose the best subset regression models more efficiently.
In Bayesian approach, a model selection method based on BIC, called Occam's Window (Raftery, 1995), has been proposed. However, when p is still too large, it is not efficient to use the Occam's Window method. For this case, preprocessing with the leaps-and-bound algorithm can be a useful way of selecting candidate models before applying the Occam's Window to construct a list of candidate models.

Simulation studies
Two possible scenarios violating standard linear regression assumptions are considered to verify performance of the model averaging methods and compare them with the result based on the standard linear regression analysis. The performance of each method has been evaluated by two criteria: (1) Risk (Clarke, 2003) is defined by (3.1) (2) Prediction Error (Breiman, 1996b) is calculated by The evaluation criteria were calculated for eight different methods for all the simulations. For the stacking, we fit five methods with different types of restrictions (Section 2.2), and denote them as S1, S2, S3, S4, and S5 respectively. Similarly, BMA is considered as the another model averaging technique (Section 2.3). As the standard linear regression analysis, we consider the full model (Full) and the model chosen from the stepwise selection method (Step).

Full
Step A set of coefficients in C1 corresponds to a model that contains three nonzero (significant) coefficients. C3 is similarly constructed as C1, while C3 is chosen to check the effect of a few large coefficients in the model averaging methods. We also consider the coefficients which have fewer number of zero coefficients. For example, C2 contains twenty coefficients that consist of sixteen ones and four zeros. C4 indicates a similar model with C2, but it includes more various magnitudes of coefficients. 100 datapoints (x i1 , . . . , x i,100 ) are generated from the standard normal distribution, independently. We also generate 100 error terms ε i from the normal distribution, but, in order to consider data with outliers, we add the five fold of the range of {ε i } (that is, δ := 5 · (max i ε i − min i ε i )) to the randomly selected 20 error terms (20% of observations) considering their signs. In other words, if ε i , i = 1, . . . , 20, is positive, we add δ to the error term, and otherwise we subtract δ from the error term. For the comparison purpose, we generate two types of datasets: (1) a dataset with error terms from the standard normal distribution; (2) a dataset with modified errors term designed as outliers as previously described. For both types of datasets, y i is calculated from (3.3), and a list of candidate models are selected based on the method suggested by Breiman (1996b). Table 1 and Table 2 show the average of evaluation metrics using datasets with no outliers and with some outliers, respectively, over hundred repetitions of the simulation. The boldface numbers in Table 1 and Table 2 indicate the best method with the minimum average risks and prediction errors for each set of coefficients. Boxplots for each simulation result are also provided in Figure 1 and Figure 2. When a dataset does not include outliers (Table 1), almost all the averages of risks for model averaging methods (S4, S5, BMA) are smaller than the full model and the model based on selected variables using stepwise selection. Particularly, BMA shows consistently better performance for all the coefficients cases than any other method. When the data contain 20% outliers, BMA shows better performance in C1 and C3. In contrast, in C2 and C4 cases (and even C1 for PEs), the stacking method (S5, and some of S4 cases) outperforms not only Full and Step, but also BMA. This can also Table 2: Averages of evaluation criteria over the 100 simulations when a dataset designed to contain outliers (20% of observations). The true model is given in (3.3), C1, . . . , C4 represent different sets of coefficients (See (3.4)), and 100 observations are considered for each simulation.
Risks (with many Outliers)

Full
Step S1 S2 S3 S4 S5 BMA Evaluation Criteria Figure 2: Boxplots of PEs and Risks over the 100 simulations when a dataset contains many outliers.

Average of an Evaluation Criterion for the Stepwise Average of an Evaluation Criterion for Each Method
be verified through Figures 1 and 2. The simulation result agrees with the findings in Breiman (1996b) in the sense that stacking shows better performance when the candidate model list contains dissimilar models, which is more likely for C2 and C4 cases because those coefficient sets contain many significant coefficients. On the other hand, since C1 and C3 contain only three significant coefficients, there are few valid candidate models to give a better performance, and the effect of combining significantly different models in stacking may be minimal.

When the distributional assumption is violated
When an error term in a linear regression model follows a heavy-tailed distribution rather than a normal distribution (e.g., t-distribution) or a skewed distribution such as a Poisson distribution, it violates the assumption of normality of error terms in the standard linear regression analysis. These error terms may have an effect on the prediction and estimation of parameters if an analysis based on the normality assumption. In particular, if the error terms follow a distribution with heavier tails than the normal distribution, the least squared method can be sensitive to a portion of data (Montgomery et al., 2012). Similarly, for the Poisson response type data, although it would be better to consider an alternative linear model such a generalized linear models (Agresti, 2013), it is possible that we conduct a standard linear regression analysis even with such type of response variable if we don't have enough information of the dataset, and it usually leads to an unreliable results. Thus, in order to deal with such a problem, especially in the prediction and estimation of the coefficients, we may consider the model averaging methods as a technique of the robust regression and compare performance with one from the standard linear regression model. Table 3: Averages of evaluation criteria over the 100 simulations when the response variable is calculated based on the error term from a heavy-tailed distribution (t 1 ). The true model is given in (3.5), T1 and T2 represent different sets of coefficients (see (3.6)), and 100 observations are considered for each simulation.
Risks (with errors from t 1 )
The result of simulation studies over 100 repetitions is given in Table 3, and corresponding boxplots are shown in Figure 3. Note that, for the illustration purpose, we scaled the unit of each metric in Figure 3 with log 10 (Evaluation Metric). According to Table 3, the stacking method under constraints S5 gives the better performance in both risks and prediction errors than the model containing all variables (Full), the model with variables selected by the stepwise method (Step), and BMA. S4 also provides good performance for Risks and PEs. In particular, S5 shows the significant outperformance not only in the sense of the averages of metrics, but also in the sense of the stability.
To evaluate performance of the model averaging methods with Poisson response data, we conduct a similar simulation by generating a dataset based on: where E[Y i ] = µ i and Y i ∼ Poisson(µ i ). Similar to the previous simulation, we use two coefficient sets (3.6) and randomly generated 100 observations (x i1 , . . . , x i8 ) from the standard normal distribution. The response value y i , i = 1, . . . , 100, is a randomly generated based on a Poisson distribution with mean e β 0 +β 1 x i1 +···+β 8 x i8 . A candidate model list is also chosen based on the Breiman's approach. Step S1 S2 S3 S4 S5 BMA

Full
Step S1 S2 S3 S4 S5 BMA log 10 Evaluation Criteria Figure 3: Boxplots of PEs and Risks over the 100 simulations when the response variable is calculated based on the error term from a heavy-tailed distribution (t 1 ). Note that evaluation metric units are modified with log 10 (x).  Table 4 shows the average of risks and prediction errors over 100 repetitions of the simulation. For both risks and prediction errors, the S5 shows better result compared to Full, Step, and even BMA. Since its performance is pretty similar to the previous simulation, we do not include corresponding boxplots, here.

Application to a water pollution data
The effect of land uses on water quality had been given a considerable attention in the 1970s. Haith (1976) collected data to investigate a relationship between land uses and water quality. The dataset consists of observations from twenty river basins in New York State. The response variable Y is the mean concentration of total nitrogen of water samples taken at regular intervals during the spring, summer, and fall months. X 1 indicates the percentage of land area in agricultural use, X 2 is the percentage of land area in forest, forest bushland, and plantations, X 3 implies the percentage of land area in residential use including urban, suburban and rural communities and strip developments with more than four residences per 1,000ft, and X 4 represents the percentage of land area in either commercial or manufacturing use. The scatterplot matrix in Figure 4 indicates that X 1 and X 2 seem to have a strong negative relationship, and there exist suspected outliers based on the scatterplot for (X 1 , X 2 ) and (X 1 , X 3 ). A linear regression model is considered to conduct a statistical analysis between land uses and the water quality. The result of the standard linear regression model analysis is shown in Table 5. Even though F-value, 9.15, for testing H 0 : β 1 = β 2 = β 3 = β 4 = 0 is very significant, all the t-values are small and none of the coefficients are significant. This is a typical indication of existence of multicollinearity among the variables. The Variance Inflation Factors (VIF) in Table 5 also supports the multicollinearity problem among X 1 , X 2 and X 3 since those VIFs are larger than 10 (Chatterjee and Hadi, 2015).
According to the result in the preliminary analysis, it is likely that one of standard linear regression assumptions is violated, i.e., strong evidence for the multicollinearity, and model averaging methods   can be an alternative way to improve the performance of analysis in the sense of prediction. Since we cannot generate a new 'test dataset' with the real application data to compare the performance of several fitting methods, Leave-one-out cross-validation estimator of prediction error (Friedman et al., 2001) was used as an evaluation metric, which is defined by wheref −i indicates the fitted model excluding the i th observation. Table 6 compares PE LOOCV for various methods. Note that we do not include the (estimated) standard error of the PE LOOCV in Table  6 because the PE LOOCV is a summary of prediction errors derived from different environments (leaveone-out based estimators) and, consequently, comparing standard errors is not likely to provide any useful information. Estimated coefficients for each method are also provided in Table 7.
In previous simulation studies, we observe that a model based on stepwise selection method (Step) usually gives the better performance than one from the model based on the all variables (Full). However, according to Table 6, PE LOOCV with the water pollution data shows that Full outperforms Step. On the other hand, considering all the methods (including model averaging methods), we see that all the five types of stacking show better results than one based on Full, Step and BMA. Table 7 shows estimated coefficients. It is known that the stepwise method with chosen variables, X 2 and X 4 here, is useful to deal with multicollinearity problem, and, accordingly, the VIF value with the new model (Y = β 0 + β 2 X 2 + β 4 X 4 ) is much less than 10 (1.05392). Since we don't have any knowledge of the true coefficients for the real-world dataset, the risk can not be used to evaluate the performance of each method. However, we are able to verify that S4 is also a useful method to select the variables so that it alleviates the multicollinearity. Therefore, we can conclude that not only the stacking method is useful to obtain better and robust performance in the prediction than one based on traditional linear regression model (including stepwise selection method) or BMA, but also it is helpful for choosing proper variables to ease the multicollinearity problem.

Discussion
The model averaging method is useful in the sense of providing a robust performance compared to a traditional linear regression analysis when some of standard regression model assumptions are violated. In this paper, two different scenarios were investigated in the paper to support the assertion. First of all, in order to verify an effect of outliers, whose existence violates one of the standard assumptions about errors in the linear regression, we compare results with a dataset having many outliers and a dataset generated with no outliers. When data do not contain outliers, Bayesian model averaging shows better performance consistently in both risks and prediction errors. On the contrary, when data include many outliers and a true model has many significant predictors, the stacking (S4 and S5) method provides better performance than Bayesian model averaging. Next, two cases are considered to represent a scenario where the distributional assumption is violated in the linear regression model. When error terms are generated from a heavy-tailed distribution (t-distribution with the degree of freedom 1), the stacking (S4 and S5) outperforms Bayesian model averaging. When a response variable is from a skewed distribution like a Poisson distribution, the stacking (S5) gives better performance in almost all cases. In the application to the water pollution data, in which some assumptions in the standard linear regression model such as collinearity are violated, stacking (S4) provides better performance than any other methods in the perspective of the leave-one-out cross-validation estimator of prediction error. Clarke (2003) argued that the stacking method shows robust performance when a true model is not on the candidate model list. Similarly, in the linear regression model, if some assumptions are violated in the model, the stacking seems to provide the robust result. On the other hand, when the true model is in the candidate models, Bayesian model averaging gives better performance. In the same manner, when the assumptions are satisfied, Bayesian model averaging always outperforms stacking. Our simulation studies show that S4 (non-negative and sum-to-one weights) and S5 (non-negative weights) provide better performances than other restriction types. Therefore, when some of standard linear regression assumptions are violated -for example, when data contains outliers, or a distribution of error terms does not follow normal distribution -S4 or S5 can be used to improve the performance in the perspective of prediction errors and risks. In particular, for the distributional assumption violation, since the stacking selects its combining weights according to differences between response values and predicted values, it will be less sensitive to the data from non-normal distribution, and it seems reasonable that the stacking method works better. However, BMA estimates each coefficient based on the BIC, and it leads to relatively poor results when likelihood functions are not correctly specified. However, the reason that S4 and S5 restrictions give better performances in the stacking method is not yet supported theoretically, so it can be left as an open question.
However, admittedly, the generalized linear model (GLM) will be more ideal model for the Poisson response data in Section 3.1. According to another simulation (which is not included in this paper), GLM (or GLM model constructed by the stepwise selection method, say, GLM-Step) always provides the best performance in terms of risks. Interestingly, for prediction errors, S5 gives slightly better performance than one from GLM (or GLM-Step). Thus, this can be another evidence that the stacking method improves performance in prediction even in such unfavorable conditions. Toward this end, we may consider model averaging methods designed for GLMs to improve performance, but it is out of scope of the purpose of this paper, so we omit them here. Also note that the performance of model averaging methods could depend on selected candidate models. In this study, since we utilized Breiman's method to construct a list of candidate models, the number of candidate models is the same as the number predictors. It is possible to include more candidate models, but our limited experience indicates that including more number of candidate models did not necessarily guarantee outperformance over the one based on the Breiman's list. One of possible reasons is that the stacking (especially for S4 and S5) and BMA method usually assigns no weights (or near zero weight) to unreliable models regardless of the number of candidate models. As long as the candidate models are well chosen, model averaging would provide a satisfying result even with the small number of candidate models. It would be worthy to conduct additional studies to evaluate the effects of various choices of candidate models. Another possible issue in regression model may occur when the error terms can not be assumed to be independent. When the independent assumption is violated, the relationship among observations can be captured with various types of time series models. If there is clear seasonality, we may use a time-series model directly, but sometimes it may not be obvious, and it has a critical effect on the precision of the OLS estimation. However, the independence assumption is related to the relationship among observations. Different approach other than model averaging may work better since model averaging methods achieve their improvement via combining various models. One possible solution might be to consider not only various candidate models, but also a modified version of the bagging (or boosting). It would be interesting to evaluate the effect of the independent assumption in model averaging methods and propose a model averaging method when this assumption is violated.
Even though we consider only a "raw" risk here, it would be fairer to compare the performance of estimating coefficients with its "standardized" version because some of methods give higher variability of risks across the simulation studies than other methods. In other words, if we can calculate the standard error of estimated coefficients based on the model averaging methods, we divide the raw risk (which is one of the evaluation metrics we consider) by its standard error, and use it as better evaluation criterion. Therefore, it is worthwhile to obtain theoretically derived standard errors in order to investigate performance of various model averaging methods more precisely. Another benefit of having standard errors is that it would allow extending the current analysis with the model averaging method to the hypothesis testing of the estimated coefficients (or weight) with corresponding standard errors. Since most of the estimators of coefficients based on the model averaging method have a complicated form, we also leave this as a future research topic.