{'fraction_non_alphanumeric': 0.03539426523297491, 'fraction_numerical': 0.007541816009557945, 'mean_word_length': 4.777825711820535, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Model Averaging Method for Supersaturated Experimental Design

In this paper, a new modified model averaging method was proposed. The candidate model construction was performed by distinguishing the covariates into focus variables and auxiliary variables whereas the weights selection was implemented using Mallows criterion. In addition, the illustration result shows that the applied model averaging method could be considered as a new alternative method for supersaturated experimental design as a typical form of high dimensional data. A supersaturated factorial design is an experimental series in which the number of factors exceeds the number of runs, so its size is not enough to estimate all the main effect. By using the model averaging method, the estimation or prediction power is significantly enhanced. In our illustration, the main factors are regarded as focus variables in order to give more attention to them whereas the lesser factors are regarded as auxiliary variables, which is along with the hierarchical ordering principle in experimental research. The limited empirical study shows that this method produces good prediction.


Introduction
Nowadays, the necessity of high dimensional data is increasing in all fields of study and has wide applications. Fortunately, the rapid growth in information technology supports the development of high dimensional data analysis. According to [1], high dimensional statistics refer to statistical inference when the number of unknown parameter is much larger order the sample size. Nevertheless, analysis of such data is impossible to be performed without additional assumption or penalization.
A typical form of high dimensional data stemmed from experimental data is supersaturated factorial design. A supersaturated factorial design is an experimental series in which the number of factors exceeds the number of runs. The supersaturated design (SSD) is recommended in high cost experiments and in situation where the number of effects is large, to screen few of a huge number of potentially active factors might be believed to be significant. There were some designs initiated to SSD. According to [2], Satterthwaite (1959), the initiator of the SSD, considered SSD to be useful in the initial investigation of large experiments and also proposed a random balance design for constructing SSD. Another construction method suggested by [3], namely E(s 2 ) criterion to measure the efficiency of SSD. Hadamard matrix was a well-known design proposed by [2] in constructing SSD. Several authors have proposed various strategies in solving the challenges in SSD data analysis using some regression procedures, such as forward selection by [4], stepwise and all-subsets regression by [5] and partial least squares by [6]. There are also shrinkage regression approaches such as Smoothly Clipped Absolute Deviation procedure [7] and the Dantzig selector [8]. Many physical experiments can be high costly and time consuming to be studied using conventional method. In computer experiments or computer-based simulation, for example, the experiments can only be simulated by a computer modelling program in which the number of factor can be large as possible. Consequently, the running time is the mostly thing to be considered instead and the goal is turning to how to produce a best prediction model at any combination of values of the inputs that fitted the data [9]. Such concerns are in accordance with the motivation of model averaging as an alternative to variable selection or screening in SSD point of view.
Model averaging (MA) is another approach to improve prediction accuracy in analysing high dimensional data. Instead of relying on only one best model, the method refers both inference and prediction to the average over a set potential models in particular manner. The key issues of the methods are for example, weight selection criterion and the way of construction the candidate model [10]. Getting the best prediction model is preferred goal instead of estimation. Application of MA in SSD, have not received many attention yet. So far, the major focus of SSD analysis has been on screening or variable selection. MA approach in SSD, based on literature study, had been conducted by [11]. The paper proposed model averaging method by using BIC weight. Another weight used in model averaging namely mallows averaging which is refer to least square model averaging proposed by Hansen [12]. In his paper, Hansen performed different approach in preparing candidate models that is by constructing nested models while the weights is selected by minimizing Mallows criterion.
The current paper introduced a new typical MA approach which is tailored to candidate models preparation. Here modification, the main factors of matrix model are categorized as focus variables that must be included in the model and their interactions that are less certain as auxiliary variables. The approach went along with effect hierarchical principle to be considered to address the relative importance and relationship among effects. The principle set the lower order effects as more important factors than the higher level effects while the same order consider to be equally likely important [4]. Henceforth, the candidate model is constructed by augmenting the matrix of focus variable with the different combination of one or several auxiliary variables. Besides reducing the number of candidate model, the reason of this differentiation is to improve the prediction performance of the model [13]. In weights selection, Mallows criterion is applied which is proven asymptotically optimal in the sense of achieving the lowest possible squared error [12].
This paper is organized as follows. In Section 2, the supersaturated experimental design is described. Section 3 reviews the intended model averaging. In the Section 4, an illustration of MA approach in SSD is performed to show how well the prediction matched the actual model. Section 6 gives some concluding remarks

Supersaturated Design
Supersaturated design is fractional factorial design in which the number of actual effects exceeded the number of runs. The SSD is divided into two classes, multi-level designs with s level on each factor and mixed-level in which each factor has different level. The current paper was concerned with multi levels of linear main effects model. Supposed the model intended is = ‫܆‬ + (2.1) where is the ݊ × 1 response vector, X is an ݊ × (݉ + 1) model matrix, vector of = (ߠ , … , ߠ ) ் constitutes the parameter and is a vector of independent normally distributed random errors with mean 0 and variance ߪ ଶ . The matrix ‫܆‬ can be a union of main effects and their interactions. It is assumed that each of the ݉ factors has two levels, contrast between high and low denoted by 0 and 1.
If the model matrix ‫܆‬ is non-singular then the matrix ‫)܆′܆(‬ ିଵ exists and the least squares estimator of parameter vector is denoted by = ‫)܆′܆(‬ ିଵ ‫′܆‬ . According to [14], if the matrix model is away from orthogonality and presence of multicollinearity then detection of the true active factors is hardly to be performed.

Model Averaging
Model averaging is considered the alternative to model selection, especially in estimation of model with high dimensional data. Model selection will choose one of a wide set of competed models based on certain selection criterion with possibly contradictory conclusion such as ܴ ଶ , BIC and AIC. On the other hand, model averaging treats each potential model as an information contributor about the parameters of interest. The contribution is expressed in a weight assigned to the candidate model which is determined based on certain criterion. The goal of model averaging is not to find the best possible model but rather to get the best possible prediction model. The first step in model averaging is to construct the model candidate. After getting estimates of each model, model averaging assigning weight to each of them and finally, takes the weighted averaging of all parameter vectors to obtain the model averaging estimator.

Framework of MA
Consider the ݊ observations ሼ(‫ݕ‬ , ‫ݔ‬ ); ݅ = 1, … , ݊ሽ, where ‫ݕ‬ is response variable and is a vector of ݉-dimensional explanatory variables where the number of predictor is allowed to increase with the sample size ݊. The relationship between the response variable and the explanatory variables will be considered from observed data. The relationship is explained in a multiple linear regression model, can be expressed as where ‫)ߝ(ܧ‬ = 0 and ‫)ߝ(ݎܸܽ‬ = ߪ ଶ . The intercept term is dropped by assuming that the means of the response variable and the input variable are already subtracted out.

Model and Estimation
Supposed the linear regression model of model in equation (2.1) is mentioned in the framework of [13] as below = ‫܆‬ + = ‫܆‬ ଵ ࢼ ଵ + ‫܆‬ ଶ ࢼ ଶ + (2.6) The matrix of explanatory variables is ‫܆‬ = ‫܆(‬ ଵ ‫܆|‬ ଶ ); ‫܆‬ ଵ (݊ × ‫)ݍ‬ is matrix of focus variables refer to intercept and main factor; ‫܆‬ ଶ (݊ × ‫)ݎ‬ is matrix of auxiliary variables refer to their interaction. Thus, vector of parameter consists of ࢼ = ൫ߠ , ߠ ଵ , ߠ ଶ , … , ߠ ൯′ and ࢼ = ൫ߠ ାଵ , ߠ ାଶ , … , ߠ ൯′. It is also assumed that ‫ݍ‬ ≥ 1 ; ‫ݎ‬ ≥ 0, ‫ݎ+‬ ≤ ݊ − 1 ; that is ‫܆‬ = ‫܆(‬ ଵ ‫܆|‬ ଶ ) has full column-rank. According to Magnus [13] the model in equation (2.6) is more general than the conventional case where typically all variables are auxiliary except the constant term. Such form also allows some variables to be included in the model even when diagnostic tests suggest it to be removed or permitted ones as must contained in the model. Moreover, such classification could reduce the number of candidate model. The least squares estimator of for a full model where all the auxiliary variables are included in model is The set of candidate models consist of different models arise when a different subset of the ࢼ ଶ , is set to zero in other words each model matrix of candidate models consist of focus variables incorporate with combination of auxiliary variables so there are totally ‫2=ܯ‬ candidate models to be consider.
Consider ܵ = ( ‫ݎ‬ × ‫)ݎ‬ a selection matrix to choose the included auxiliary variables, where ‫ݎ‬ ݅ is the number of auxiliary variables contained in ‫ܯ‬ . So, the estimators for the candidate model ‫ܯ‬ , is If the corresponding prediction candidate model is then model averaging prediction of equation (2.6) can be written as

Mallow Weight Criterion
There are various methods in determining the weights of model averaging. One of them was Mallow averaging proposed by Hansen [12] in which the selection weights is performed by minimizing a Mallows criterion. Such criterion for the averaging estimator is If ‫)(ܲ(ݎݐ‬ is rewritten as = (݇ ଵ , … , ݇ ெ )′; ݇ is the number of parameters in ith model, then () = ᇱ ො ᇱ + 2ߪ ଶ ′ (2.9) which is a quadratic form in the vector . By minimizing the criterion in (2.9), the weight vector is ෝ = argmin ∈ ℋ ‫ܯ‬ () (2.10) In order to apply model averaging method in SSD, an illustration will be presented based on [6] on pg. 443. Its goal was to develop an epoxide adhesive system for bonding a polyester cord. The experiment involved 23 factors with 14 runs or experiments. The response is strip adhesion, which needs to be maximized. After a screening analysis using stepwise regression, there are 6 significant main effects to be considered. Those effects are 4,10,12,15 and 19. A further analysis was planned by involving also two-factor interaction effects. As there are 5, hereafter denoted by A,B,C,D and E respectively, the analysis would consist of 5 main effects and 10 interaction effects with an additional parameter of intercept. Thus, there were as many as 16 parameters to be estimated with only 14 observations that was a 14 × 16 design matrix. It was a typical problem of high dimensional data, typically, supersaturated design experiment that reasonable to be solved using model averaging analysis. Based on those resulted factors, the further analysis conducted by using model averaging with the follow steps.

Conclusion
This study concludes that model averaging method could be applied in supersaturated experimental design as a typical of high dimensional data. That method was a modification of model averaging in which the candidate model construction is performed by distinguishing the matrix model into focus variables and auxiliary variables whereas the weighted selection is using Mallows criterion. In the case of experimental data set, main effects are regarded as focus variables, while the interactions as auxiliary variables. This differentiation follows the hierarchical ordering principle in experimental researches to address the relative importance among effects.
Based on our limited empirical study as was illustrated above, this method results a prediction model with good performance according to the value of MAPE and ܴ ଶ . However, an advance study is needed to assess its performance in handling multilevel and mixed level experiments. Comparison with other type of model averaging is also highly recommended to be done in order to evaluate whether the modified model averaging method is better than others. Moreover, the further study in concern with criterions of estimator such as unbiasedness, efficiency and consistency would be a worth complement task.