{'fraction_non_alphanumeric': 0.03913978494623656, 'fraction_numerical': 0.012405206564799095, 'mean_word_length': 4.436377061284765, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
High-dimensional influence measure

Influence diagnosis is important since presence of influential observations could lead to distorted analysis and misleading interpretations. For high-dimensional data, it is particularly so, as the increased dimensionality and complexity may amplify both the chance of an observation being influential, and its potential impact on the analysis. In this article, we propose a novel high-dimensional influence measure for regressions with the number of predictors far exceeding the sample size. Our proposal can be viewed as a high-dimensional counterpart to the classical Cook's distance. However, whereas the Cook's distance quantifies the individual observation's influence on the least squares regression coefficient estimate, our new diagnosis measure captures the influence on the marginal correlations, which in turn exerts serious influence on downstream analysis including coefficient estimation, variable selection and screening. Moreover, we establish the asymptotic distribution of the proposed influence measure by letting the predictor dimension go to infinity. Availability of this asymptotic distribution leads to a principled rule to determine the critical value for influential observation detection. Both simulations and real data analysis demonstrate usefulness of the new influence diagnosis measure.

Influence diagnosis is important since presence of influential observations could lead to distorted analysis and misleading interpretations. For high-dimensional data, it is particularly so, as the increased dimensionality and complexity may amplify both the chance of an observation being influential, and its potential impact on the analysis. In this article, we propose a novel high-dimensional influence measure for regressions with the number of predictors far exceeding the sample size. Our proposal can be viewed as a high-dimensional counterpart to the classical Cook's distance. However, whereas the Cook's distance quantifies the individual observation's influence on the least squares regression coefficient estimate, our new diagnosis measure captures the influence on the marginal correlations, which in turn exerts serious influence on downstream analysis including coefficient estimation, variable selection and screening. Moreover, we establish the asymptotic distribution of the proposed influence measure by letting the predictor dimension go to infinity. Availability of this asymptotic distribution leads to a principled rule to determine the critical value for influential observation detection. Both simulations and real data analysis demonstrate usefulness of the new influence diagnosis measure.

1.
Introduction. An observation is flagged influential if some important features of the analysis are substantially altered after this observation is removed [13]. Presence of influential observations would possibly lead to distorted analysis and misleading results [18], and therefore it is important to be alert to influential observations and take them into consideration when interpreting the results. In the classical normal linear model setup, regression coefficient estimate was chosen, naturally, as the feature whose substantial change defines influential observations. Toward that end, [12] proposed a difference measure between the OLS estimate on the full data and that on the subset of data without the observation in question. This measure, which is later on referred in the statistical literature as the Cook's distance, quantifies the contribution, or influence, of individual data observation on the regression coefficient estimate. Consequently an observation with a large Cook's distance is deemed as influential. Since its introduction, the Cook's distance has been routinely employed in regression analysis, due to its clear interpretation from the case deletion point of view, and its easy computation without having to re-estimate the model for each removed observation. The topic is covered in most standard regression textbooks, and it is implemented in popular statistical software such as R and SAS.
Thanks to the aforementioned works, substantial insights have been gained on influence diagnosis. However, it is important to note that, all existing diagnosis approaches have been developed under the assumption that the number of predictors in regression is fixed. As such, none is immediately applicable to high-dimensional regression analysis, where the number of predictors p far exceeds the sample size n. On the other hand, nowadays prevailing in both science and business are data with unprecedented size and dimensionality, calling for the development of high-dimensional influence diagnosis. Detection of influential observations in high-dimensional data analysis, in our opinion, is equally, or to some extent, even more important than in a classical setup. This is partly because the increased dimensionality and complexity of the data may amplify both the chance of an observation being influential as well as its potential impact on the analysis. Moreover, the peculiar data observations themselves may be of practical importance in addition to data modeling. The diagnosis task, nevertheless, is more challenging in high-dimensional data analysis, and is far from a direct extension of existing diagnosis approaches. To the best of our knowledge, influence diagnosis in a high-dimensional setting has received little attention despite its evident importance.
The first challenge is the definition of influential observation. In other words, which feature of the analysis should one choose such that its substantial alternation defines an influential observation? In the classical setup, an observation is deemed influential if it incurs serious change in regression coefficient estimate. In high-dimensional regression where p > n, the ordinary least squares estimator is highly unstable as the gram matrix is not invertible. On the other hand, we recognize that variable selection and variable screening are of particular importance in high-dimensional regression analysis. There has been a vast literature on variable selection in recent years, including the LASSO [34], the adaptive LASSO [36,40,45], the SCAD [21], the bridge estimator [24,26], the LARS algorithm [19], the Dantzig selector [8], the sure independence screening rule [22], SIS, the forward regression [35], FR, among many others. Underlying all those selection methods, one statistic plays a critical role and, that is, the marginal covariance, or equivalently, marginal correlation between the response and the individual covariates. To clarify, we note that, SIS is directly defined based on this statistic, whereas the first step of the forward regression hinges on the estimated marginal covariance too. In addition, the sample marginal covariance, in addition to the Gram matrix, is an important input for the well celebrated LARS algorithm, as well as the LASSO, the adaptive LASSO and the Dantzig selector.
Motivated by this vital observation, we choose the marginal correlation as the feature that defines influential observation. We propose a new influence diagnosis measure, which continues to utilize the leave-one-out idea of the classical Cook's distance, but is based on the combined marginal correlations between the response and all predictors. The new measure is applicable to high-dimensional setting where p > n, and is very fast and easy to compute. Unlike the classical Cook's distance that quantifies the individual observation's influence on the least squares coefficient estimate, the new measure captures the influence on the marginal correlation, which in turn exerts serious impact on variable selection and other downstream analysis. The choice of the marginal correlation as the defining feature of our influence diagnosis does not imply that the marginal correlation is our ultimate goal of interest. Instead, it reflects influence on important analysis features including parameter estimation, variable selection and screening. This definition of influential observation in a high-dimensional setting can be viewed as our first contribution. 4

ZHAO, LENG, LI AND WANG
Our second contribution is that the explicit asymptotic distribution for the proposed influence measure is derived. Availability of this asymptotic theory offers a principled guidance to determine the critical value for the influence measure. Subsequently, we propose a false discovery rate based procedure for that purpose [5,6]. We remark that, in the classical setup where p is fixed, a standard Taylor's expansion type analysis [12] revealed that the classical Cook's distance's major variability is due to the observation under investigation and its sample size is only one. This rules out the possibility of establishing a standard asymptotic theory for the classical Cook's distance. To determine an appropriate threshold value for the classical Cook's distance, its distribution can be obtained by bootstrap if the true model is a parametric linear model. However, such a bootstrap procedure requires a parametric model assumption and can be computationally expensive especially for high-dimensional data. By contrast, the asymptotic distribution of the proposed influence measure is attainable in our setup, since the predictor dimension goes to infinity along with the sample size, and the threshold is easy to obtain.
When facing high-dimensional data diagnosis, an intuitive solution is to continue using the classical Cook's distance but to replace the OLS coefficient estimate with a regularized estimate, for instance, a LASSO estimate. This modified Cook's distance approach could be particularly useful when data perturbation concentrates on the nonzero coefficients, as it avoids unnecessary variability caused by irrelevant covariates. However, it also has several limitations. First, this solution interweaves influence diagnosis with variable selection, which can be flawed if the influence is reflected on variable selection itself. For instance, an influential observation may substantially alter the chosen tuning parameter of the LASSO, resulting in a totally different regularized coefficient estimate, which in turn affects the modified Cook's distance. Second, the tuning parameter of the LASSO, in principle, should be updated for every reduced data set, and this re-estimation requirement can be very expensive computationally, especially when the regression dimension p is large. Third, the asymptotic properties of the modified Cook's distance seem intractable analytically, which makes the thresholding of influential data difficult, whereas a bootstrap alternative to choose the thresholding value is again computationally expensive. Moreover, while there exist many competing variable selection methods, it is unclear which selection method is the best choice in the context of influence diagnosis. By contrast, our influence measure is not constrained by any particular variable selection method, and this flexibility could benefit downstream analysis. In Section 3, we carry out an intensive numerical study to compare this modified Cook's distance with our proposal, and this detailed comparison can be viewed as the third contribution of this article. variable screening (c), as the perturbation parameter κ varies. "Before HIM" denotes the analysis on the full data, and "After HIM" denotes the analysis on the reduced data after removing the influential observations flagged by our proposed high-dimensional measure (HIM).
Before we proceed, we quickly show a simulated example to illustrate two points: first, how various aspects of a high-dimensional regression analysis, including regression coefficient estimation, variable selection and variable screening, can be seriously affected by influential observations, and second, how our proposed measure can help limit such influence. The data was generated from a linear model with p = 1000 predictors, n = 100 observations, among which 10 observations were influential. The magnitude of the influence was dictated by a scalar κ with a larger value indicating a larger influence. More details can be found in the setup of model 1 in Section 3. Evaluations include error in coefficient estimation, error in variable selection after applying the LASSO [34], and error in variable screening after applying the SIS [22]. The results are averaged over 200 simulation replicates, and are reported in Figure 1. It is clearly seen from the plot that, influential observations could have drastic effects on various features for high-dimensional data analysis. Meanwhile, our marginal correlation based diagnosis could greatly help control the adverse effects after detecting and removing those influential data points.
The rest of the artlicle is organized as follows. Section 2 begins with a review of the classical Cook's distance, then presents our new high-dimensional influence measure, along with a comparison with the Cook's distance, the asymptotic properties and a power study. Section 3 includes an intensive 6 ZHAO, LENG, LI AND WANG simulation study and a microarray data analysis. Section 4 presents a generalization of our proposal from the normal linear model to the generalized linear model. Section 5 concludes the paper with a discussion. All technical proofs are given in the Appendix and the supplementary material [41].
2.1. Linear models and classical Cook's distance. In this article, we focus on influence diagnosis in the context of the classical linear regression model. Meanwhile, we note that the proposed idea can be readily extended to a much broader class of regression models, and we will discuss one such extension in Section 4. Consider the following model: . . , Y n ) ⊤ denotes the n × 1 response vector, and X denotes the n × (p + 1) design matrix with the ith row being p + 1 dimensional vector (1, X ⊤ i ), i = 1, . . . , n. To quantify the influence of the kth observation on regression, 1 ≤ k ≤ n, [12] employed the leave-one-out idea by studying the OLS estimate of β while the kth observation is excluded from estimation. That is, one minimizes the is the (n − 1) × 1 response vector with Y k removed, and X (k) is the (n − 1) × (p + 1) design matrix with the kth row X k removed. Cook [12] naturally chose the estimate of β to define influence, and intuitively, if an observation is influential, the difference betweenβ andβ (k) is expected to be large. This leads to the following discrepancy measure, that is, the Cook's distance: In the high-dimensional regression setting, the classical Cook's distance (2.2) encounters some difficulties. When p is close to n, the OLS estimate is known to be unstable, which would in turn cause D k to be unstable. When p > n, the classical Cook's distance is not directly computable, because the OLS estimatorβ becomes unstable. For those reasons, the regression coefficient estimate may no longer be the best choice to define influence in high-dimensional analysis. This motivates us to consider an alternative influence measure for high-dimensional data.

2.2.
High-dimensional influence measure. In high-dimensional regression analysis where p ≈ n or p > n, variable selection (screening) plays a central role, whereas marginal covariance or correlation is crucial to the majority of variable selection approaches. Motivated by this observation, for highdimensional data influence diagnosis, we choose marginal correlation, instead of regression coefficient, as the feature that defines influence. Individual observation's influence on marginal correlation is to transmit to various features of downstream analysis, such as variable selection and coefficient estimation.
More specifically, we first define the marginal correlation as We then obtain the sample estimate, . . , p, whereμ xj ,μ y ,σ xj andσ y are the sample estimates of µ xj , µ y , σ xj and σ y , respectively. Next, we continue to use the leave-one-out principle as in the classical Cook's distance case, and compute the marginal correlation with the kth observation removed aŝ y are the corresponding sample estimates with the kth observation removed. Finally, we define the influence measure based on the marginal correlation as We refer to D k as the high-dimensional influence measure, or HIM for brevity. We make a few remarks. First, we note that the marginal correlation can be easily computed regardless of the predictor dimension, and such computational advantage is practically very useful for high-dimensional data analysis. Second, the proposed influence measure is built upon the marginal correlation coefficient, and is effectively scale invariant. However, it does not imply that marginal correlation is the ultimate feature of interest in our influence diagnosis. Instead, a substantial change on the marginal correlation caused by a data point is to exert influence on important features such as variable selection and parameter estimation, as we have seen in Figure 1. As such, for an estimation method to be robust to unexpected perturbation [15,42,43], the sample marginal correlation should be sufficiently robust. This is an important and necessary condition, although not necessarily sufficient. Finally, use of the marginal correlation to define the influence measure does not imply that we assume a marginal model. Instead, we still assume the joint model (2.1). As it may seem unclear how a marginal measure can capture the influence for a joint model, we will demonstrate through a simple joint model later in Section 2.5 that, the newly defined D k can indeed identify the influential observation with probability one. This use of marginal correlation is also similar in spirit to the sure independence screening procedure for a joint normal model [22], but is in a different context. Fan and Lv [22] use marginal correlation for the variable screening purpose, while we use it for influence diagnosis.
The proposed high-dimensional influence measure also shares some similarity as the classical Cook's distance. Note that the Cook's distance can be reformulated as is an increasing function of both |ǫ k | and h kk . As such, an observation has a large value in Cook's distance, if it has a large residual or it is a high leverage point in terms of h kk . Our proposed information measure shares a similar spirit. In Section 2.3, we will derive a decomposition of our influence measure D k under some conditions, and will show that D k is mainly dominated by a term called B 2 , which is of the form Consequently the kth data point (X k , Y k ) is more likely to be marked influential, if it has a large response and a large value of X k 2 . Here X k 2 plays a similar role as h kk in the classical Cook's distance, for detecting influential points induced mainly by covariates, whereas Y k plays a similar role as the residual in the Cook's distance, for detecting the influential point induced by abnormal responses.
2.3. Theoretical properties. We next establish the asymptotic distribution of the proposed high-dimensional influence measure D k as both the sample size n and the dimensionality p go to infinity. Toward that end, we impose the following conditions.
(C.1) For any fixed j = 1, . . . , p, ρ j is constant and does not change as p increases.
3) The predictor X i follows a multivariate normal distribution and the random noise ε i follows a normal distribution.
Condition (C.1) is very general, since it only requires that for any fixed j, ρ j is a constant independent of p. A sufficient condition for condition (C.2) to hold is that all eigenvalues of Σ are finite. This condition also permits eigenvalues of Σ to diverge to infinity but at a slower rate compared to the dimensionality. The normality assumption on X is mainly for convenience, and can be relaxed, for instance, to distributions with sub-Gaussian tails, at the expense of more lengthy proofs. In addition, since the error term is assumed normal, Y is normally distributed.
Next, we derive a decomposition of D k , that is, to serve as a basis for its asymptotic distribution. The result is presented in a way such that µ y , µ xj are assumed to be 0 and σ xj , σ y are 1 for 1 ≤ j ≤ p. This leads to simplified On the other hand, we note that this standardization is only for the purpose of simplifying the presentation and it loses no generality. As we will show later in Proposition 2, replacing the unknown quantities µ xj , µ y , σ xj and σ y with their consistent sample estimates would not alter D k 's asymptotic distribution. For t, s = 1, . . . , n, let K p,ts = j X tj X sj /p and c p = max 1≤j≤p λ j . After some algebraic computation, we obtain that Then we have the following result on the expectation of D k along with the variance of its decomposition in terms of B's. Now we return to the asymptotic distribution of D k . Proposition 1 helps to derive the asymptotic distribution of D k . We first present the result assuming µ xj , µ y , σ xj and σ y are all known. Then we obtain the asymptotic distribution when µ xj , µ y , σ xj and σ y are replaced by their sample estimates.
When there is no influential point and min{n, p} −→ ∞, we have where χ 2 (1) is the chi-square distribution with one degrees of freedom.
Next, we consider the asymptotic distribution of D k when µ xj , µ y , σ j and σ y are unknown. A natural choice is to replace them by their corresponding sample moment estimates asμ . Another choice is to employ robust estimators, for example, the median in place of the mean, and the median absolute deviation in place of the standard deviation. The following proposition shows that the conclusion of Theorem 1 continues to hold as long as u xj , u y , σ xj and σ y are replaced by √ n-consistent estimates under certain moment assumptions. LetẎ t = (Y t − µ y )/σ y ,Ẋ tj = (X tj − u tj )/σ tj , t = 1, . . . , n, j = 1, . . . , p and (Q xj , R xj ) = ((μ xj − µ xj )/σ xj , σ xj /σ xj ) and (Q y , R y ) are defined similarly. Furthermore, let S Qx = lim sup n→∞ E(n 1/2 Q x1 ) 8 , S Rx = lim sup n→∞ E[n 1/2 (R x1 − 1)] 8 , S Qy = lim sup n→∞ E(n 1/2 Q y ) 8 and S Ry = lim sup n→∞ E[n 1/2 (R y − 1)] 8 . We make the following additional assumption.
(C.4) For all 1 ≤ j ≤ p, (Q xj , R xj ) are the same symmetric function of {Ẋ tj , for t = 1, . . . , n}; and (Q y , R y ) are also the same symmetric function ofẎ t for t = 1, . . . , n. We assume that S Qx , S Rx , S Qy and S Ry are finite. . . . , x p )) and f 1 and f 2 are symmetric functions. Condition (C.4) is a mild condition. Recall that (X i , Y i ), i = 1, . . . , n are i.i.d. normal in Theorem 1. Whenμ xj ,σ xj are the moment estimates, we have Q xj = n −1 1≤t≤nẊ tj ∼ N (0, 1/n) and consequently S Qx is finite. Moreover, we have R xj = 1/S nj where S 2 nj is the sample variance of {Ẋ tj , t = 1, . . . , n}. Since S 2 n1 ∼ χ 2 n−1 /(n − 1), it is easy to verify that S Rx is also finite. Similarly, S Qy and S Ry are also finite with moment estimatesμ y andσ y . Under the normality of (X, Y ), (C.4) also holds for some robust estimates.
We remark that the asymptotic distribution of the high-dimensional influence measure D k is obtained as the number of predictor p goes to infinity. This is different from the case of classical Cook's distance D k where p is fixed, for which a standard asymptotic distribution is not attainable. We view this as a blessing of dimensionality in contrast to the usually conceived curse of dimensionality. For more examples of blessing of dimensionality, see [17] and [28].
2.4. Influence diagnosis. An important implication of Theorem 1 is that we can now obtain a p-value for influence diagnosis. Specifically, for the hypothesis that the kth observation is not influential versus its alternative, the p-value is P (χ 2 (1) > n 2 D k ). Given that the number of predictors p is usually large and multiple hypotheses are tested simultaneously, we employ the false discovery rate based multiple testing procedure of [5] to determine which hypothesis should be rejected while controlling the family-wise error. Denote n infl as the number of influential observations among the n observations, n tp and n fp as the number of the observations that are correctly rejected and incorrectly rejected, respectively, and r as the total number of rejections in the n hypotheses testing. Then the power and the false discovery rate are denoted as Power = n tp /n infl and FDR = n fp /r, respectively. We will set FDR level being small, such as 0.05, and report the power and other quantities in the numerical study section. We also remark that more sophisticated alternative multiple testing procedure, for example, in [6], [20] and [32], can be used in conjunction with our approach, but, that is, not the focus of this article.

2.5.
A power comparison of two influence measures. We next study the power property of both the new diagnosis measure and the Cook's distance via a simple model. This study serves two purposes. First, we can gain insight about difference between the two diagnosis measures. Second, it offers evidence that the marginal correlation based measure is capable of detecting influential observation in a joint model with a large probability.
More specifically, we consider the model (2.1), but drop the intercept for simplicity. The predictors X i , i = 1, . . . , n, are i.i.d. observations from a multivariate normal distribution N p (0, Σ) where Σ is a p × p covariance matrix with all its diagonal elements σ jj = 1. The error term ǫ i is of the structure ǫ i = e i + c i , where e i follows a standard normal distribution and c i is constant, c 2 = · · · = c n = 0. Under this setup, the first observation is an influential point as long as c 1 = 0, and we aim to establish the power of both the classical and our proposed high-dimensional influence measure in identifying this influential observation. Let D i be the Cook's distance defined in (2.2) for the ith observation, D  The proof is given in the supplementary material [41]. Here we compare the two sets of conditions to gain some insight about the difference of the two diagnosis measures. First, we examine the condition max{n −1 p 6 , |c 1 | −1 n 2/3 } → 0, that is, required by the Cook's distance. The condition |c 1 | −1 n 2/3 → 0 here is to ensure that the influence of the first observation does not vanish as n goes to infinity. Moreover, in terms of the predictor dimension p, the classical Cook's distance is defined when p < n. Consequently, the condition n −1 p 6 → 0, or equivalently, p = o(n 1/6 ), constrains the growing rate of p with n at a much slower rate. We note that although this rate may not be the optimal one, the condition p = o(n) is clearly necessary for the classical Cook's distance to be feasible. Next, we examine the condition max{|c 1 | −1 (log n) 1/2 , l p p −2 c −4 1 n} → 0, that is, required by our new influence measure. For illustration, we consider a simple case with all the eigenvalues of Σ bounded and p > n. We know immediately that both l p /p and n/p are bounded. Accordingly, we should have l p p −2 c −4 1 n → 0 as long as c 1 → ∞. As log n → ∞ when n → ∞, then a sufficient condition for max{|c 1 | −1 (log n) 1/2 , l p p −2 c −4 1 n} → 0 is that (log n) 1/2 /|c 1 | → 0. This suggests that the influence point can be consistently detected, as long as c 1 diverges to infinity at a speed faster than (log n) 1/2 . This is clearly a rate much slower than n 2/3 . Finally, the bounded eigenvalue condition (C.5) is commonly used in the literature for estimating covariance matrices [7]. Here it is assumed for the Cook's distance case. For the new diagnosis measure, (C.2) is required instead, which is weaker than (C.5).
3. Numerical studies. We have carried out an intensive simulation study, along with a microarray data analysis, to examine the empirical performance of our proposed high-dimensional influence measure. Since the classical Cook's distance depends on both leverage points and outliers, in our simulation study, we consider three different scenarios where there exist outliers only (model 1), leverage points only (model 2), or mixed leverage points and outliers (model 3). For the scenarios with leverage points (models 2 and 3), we further consider sub-scenarios where important covariates contribute to leverage observations, or noisy covariates contribute to leverage observations. Below we present the summary of the analysis.

Simulation models.
For all simulations, we set the sample size n = 100, and the number of predictors p = 1000. We set 10% of total observations as influential, so thatñ = 10. We consider the model where X i is multivariate normal with cov(X ij , X ij ′ ) = 0.5 |i−j| , ε i follows the standard normal distribution, and β = (3, 1.5, 0, 0, 2, 0, . . . , 0) ⊤ . We simulated n = 100 i.i.d. observations from this model. Next, we reset the first n = 10 data observations as coming from another model, where perturbations are to be introduced on the regression coefficient, the covariates and their combination. In particular, we have considered three perturbation models of generating influential points.
Model 2. The perturbation was introduced on the predictors and keep the response uncontaminated. That is, for i = 1, . . . ,ñ,Ỹ i = Y i andX ij = X ij + 30κI {j∈S} , j = 1, . . . , p. In other words, a set S of predictors admit a different pattern, and its magnitude is controlled by the scalar κ. We examined three choices of S: S 1 = {1, . . . , 100}, and in this case, the influenced predictors overlap with those truly relevant ones {1, 2, 5} in β; S 2 = {p − 100, . . . , p}, and as such there is no overlap; and S 3 = {1, . . . , p}, and in this case, all predictors are subjected to potential influence.
It is clear that κ is the parameter that dictates the magnitude of the influential points. When κ = 0, there is no influential point. We used κ = 0, 0.4, 0.8, 1.2 and 1.6 in our experiment.
3.2. Performance evaluation. We evaluate and compare our proposed influence measure in several ways. First, we study the potential impact of influential data and how the proposed diagnosis measure could help limit such impact. Toward that end, we first applied the LASSO or SIS to the full data. Then we computed the proposed high-dimensional influence measure, evaluated the corresponding p-value, and applied the multiple testing procedure of [5], with the false discovery rate fixed at α = 5%. We then obtained a reduced data set by removing those flagged influential points and applied the LASSO or SIS to the reduced data set. We evaluated the impact of influential data in terms of coefficient estimation, variable selection, and variable screening. For coefficient estimation, we report the error between the estimated and true β, ERR = β − β true 2 ; for variable selection, we report the false positive rate, FPR = #False Positive/#True Negative; and for variable screening, we report the coverage probability CP. In addition, we also report the empirical power of our influence identification procedure.
Second, we compare our method to two potentially competing solutions in high-dimensional influence diagnosis. One is a modified Cook's distance based on the LASSO. That is, we continue to employ the classical Cook's distance, but estimate the regression coefficient β under a LASSO penalty and as such avoid the difficulty of the OLS estimate when p > n. This seems a very natural solution. We compare it with our proposal in terms of estimation accuracy, selection accuracy and power. On the other hand, we note the lack of asymptotic theory for this modified Cook's distance. To determine the threshold for influential data, one may use bootstrap. However, in our comparison, we simply label the observations with the largestñ modified Cook's distance as influential. This is not feasible in practice, but provides a useful benchmark for comparison. The other competing solution is the penalized least absolute deviation via the LASSO penalty (LAD + LASSO) [4,37]. Due to the use of the least absolute deviation as the loss function, this method is designed to handle heavy tailed errors in linear regression, and as such a potentially useful way to limit impact of the influence observations.  Tables 1-3. We make the following observations.
(1) First, the presence of influential points significantly affects variable selection and screening accuracy. This can be seen by comparing the re- sults between SIS and SIS + HIM in terms of CP. Consider, for example, Table 1. As κ increases, the coverage probability of the SIS method deteriorates quickly from 1 with κ = 0 to 0 with κ = 1.6. This confirms that influential observations do affect variable screening consistency. Meanwhile, the performance of SIS + HIM is quite encouraging as its CP values maintains at 1 for every κ value considered. This suggests that the proposed HIM method helps SIS in removing the influential observations.
(2) Second, the presence of influential observations does affect estimation accuracy seriously. This can be seen clearly by comparing the results of LASSO and LASSO + HIM in terms of ERR values. For instance, the ERR values in Table 3 for LASSO with S 1 increases quickly from 0.446 with κ = 0 to 14.498 with κ = 1.6. This confirms that influential observations do affect the accuracy of the LASSO estimate in a negative way. However, we find that the ERR values of LASSO + HIM are always well controlled with ERR < 1.5. In fact, as κ increases, the power for HIM to detect influential observation increases. Thus, those influential observations are more likely to be detected and eliminated from the data analysis. This makes the ERR values of LASSO + HIM eventually converges to a level around ERR ≈ 0.5, as κ increases. This confirms the usefulness of the HIM method for LASSO estimation, even though its definition only involves marginal correlation coefficients.
(3) Third, the performance of LASSO + CD is mixed. If the perturbation is due to the response only as in Table 1, it does yield much better performance than LASSO with much smaller ERR values. This suggests 16 ZHAO, LENG, LI AND WANG that LASSO + CD can perform well to limit the effect of influential points. However, even for this example, it is still outperformed by LASSO + HIM. However, the story changes if the perturbation is due to the predictors as in Table 2. This is to be expected because, with contaminated predictors, LASSO is no longer a stable method for variable selection. If predictors are selected incorrectly, the subsequent modified Cook's distance cannot be calculated appropriately. This makes the performance of LASSO + CD unsatisfactory.
(4) Fourth, as a robust regression method, we find that LAD + LASSO performs quite well. Its ERR values are smaller than those of the LASSO estimates in all the tables. However, in most cases, it is still outperformed by LASSO + HIM as seen from Tables 1 and 3. (5) Lastly, we find that for most cases, the reported FPR values are well controlled. Furthermore, as κ increases, the corresponding empirical power increases toward 100%. These findings are consistent with the theoretical claims in Theorems 1 and 2.
To summarize, our simulation experiments confirm that the proposed HIM method is useful in controlling the effects of the influential observations in terms of parameter estimation and variable screening.

3.4.
A real data example. We applied our proposed influence diagnosis approach to a microarray data of [31], and noted that the analysis results become substantially different when the detected influential observations are removed. For this dataset, F1 animals were intercrossed and then 120 twelve-week-old male offspring were selected for tissue harvesting from the eyes and for microarray analysis. The Affymetrix microarrays that were used to analyze the RNA from the eyes of those F2 animals contain over 31,042 different probe sets. Among them, one probe is for gene TRIM32, which was recently found to cause Bardet-Biedl syndrome [10], a genetically heterogeneous disease of multiple organ systems including the retina. One goal of interest of this data analysis is to find genes whose expressions are correlated with that of gene TRIM32. We first followed [27] to exclude probes that were not expressed in the eye or that lacked sufficient variation, which results in 18,975 probes as regressors. We then followed [22] to retain the top 1000 probes that are mostly correlated with the probe of TRIM32. The resulting analysis has p = 1000 predictors and a sample size n = 120. As a standard procedure [27], all the probes are standardized to have mean zero and standard deviation one.
We next applied our method with FDR rate α = 0.10 to the data, and identified a total of 5 influential observations. Their corresponding p-values were 0, 0.0004, 0.0011, 0.0029 and 0.0033, respectively. We also show the logarithm of p-values versus the indices for all 120 observations in Figure 2. To assess the influence of the detected points, we again compared the LASSO estimate with and without those points. Since we used ten-fold cross-validation to select the tuning parameter and every run is random, we repeated this analysis 100 times and report the average results.
We summarize the difference of the estimates in the following aspects: the sparsity, the norm difference and the angle between the two estimates. First, by removing the identified influential observations, the resulting LASSO estimate is considerably more sparse. The average model size with the full data is 63. By contrast, the average model size without the influential observations reduces to 27 on the average. The existence of the potential influential points clearly shows a significant effect on the model size. Besides that, the average number of the commonly selected predictors by fitting the full data and the reduced data, respectively, is only 8.67, which again shows clear discrepancy of the two estimates. Consequently, the influential points identified by our approach seem to have significant effect for subsequent analysis. Second, denote d 0 = β full 2 , d 1 = β redu 2 and d 2 = β redu −β full 2 , whereβ full is the LASSO estimate using all the observations andβ redu is the estimate after removing the influential points identified by HIM. We observe that the average of (d 0 − d 1 )/d 0 is 0.532 and that of d 2 /d 0 is 0.972. Both show that the estimates without influential points are quite different in terms of the ℓ 2 norm. In addition, the angle betweenβ full andβ redu , which is defined asβ ⊤ fullβredu /d 0 d 1 , equals 0.262, averaged over 100 times. These numbers again indicate that the estimates change substantially after removing the influential observations. In summary, this analysis illustrates the importance of influence diagnosis, and the identified influential observations should be treated with extreme care.
4. Extension to generalized linear models. The main idea of the highdimensional influence measure can be extended to a broad class of regression models. Here we briefly discuss one such extension to generalized linear models (GLM). Assume that the data (X i , Y i ) follow an exponential distribution with the canonical probability density function, f (y; θ) = exp{yθ − b(θ) + c(y)}, and the conditional mean is of the form , where g is a known link function. For the purpose of feature screening in ultra high-dimensional regressions, [23] introduced a marginal utility measure, the maximum marginal likelihood estimator, aŝ That is,β j is the maximum likelihood estimator of fitting a GLM model of Y on the jth predictor X j alone plus an intercept. As remarked by [23], this measure can be rapidly computed.
In the context of high-dimensional diagnosis, we define the high-dimensional influence measure for generalized linear models for the kth observation, k = 1, . . . , n, as denotes the maximum marginal likelihood estimator but with the kth observation removed. For GLM, the estimatorβ j andβ (k) j may not have a closed-form solution. Consequently, the exact distribution of the proposed statistic D glm k is complicated and some approximation is necessary. The detailed derivation, however, is beyond the scope of this paper. In practice, one can always sort the values of {D glm k , k = 1, . . . , n} and remove those observations associated with large values of D glm k . We have conducted a small simulation study to examine the empirical performance of this measure for GLM. The simulation setup is similar to that of model 1 in Section 3.1, except that this time we adopt a binary response model, P (Y i = 1 | X i ) = 1/[1 + exp{−(2 + X ⊤ i β)}], where β = β true = (5, 5, 0, . . . , 0) ⊤ , and the outliers are generated from the model β = β infl = (5, 5, 0, . . . , 0, −κ, . . . , −κ) ⊤ with p/2 many κ's. We set n = 100, with 10% influential observations, that is, n infl = 10, and we set p = 50 or 100. Since the asymptotic distribution of D glm k is not available for the logistic regression, we flag the 10 observations with the largest p-values of D glm k as influential. For a binary response, one is often interested in classification. As such we compare the misclassification error rate for the full data as E full and for the reduced data as E redu without the detected influential points. We also report the empirical power. The results out of 200 data replications are summarized in Table 4. From Table 4, we note that the proposed method has some power for a logistic model, but it is lower than that in a linear model. This is probably due to the fact that a binary response contains much less information, and thus detecting influential observations in a logistic model is much more challenging, especially in a high-dimensional setting. On the other hand, we also observe from Table 4 that removing those points with the largest values of D glm k improves the classification accuracy by a large margin. This again suggests that the usefulness of influence diagnosis. Meanwhile, we remark that further investigation into both theoretical and empirical properties of high-dimensional influence measure in GLM is warranted.

Conclusion.
We perceive several future avenues to extend the proposed work in this article. First, we have employed the leave-one-out principle when quantifying influence of individual observations. We expect that our high-dimensional influence measure can also be generalized to the cases of leaving out pairs of observations, or triplets or more. Such a strategy can be useful when those observations conceal one another [18]. Second, we have focused on the classical linear model in our development, while extension to more sophisticated models, such as the generalized linear model, that is, briefly examined in Section 4, survival models, and semiparametric additive models, deserve further investigations. Finally, our proposal deals with the cross sectional data with i.i.d. observations. It is interesting to extend the proposed influence measure to complex correlated data such as longitudinal data where dependence among observations needs to be taken into consideration in influence diagnosis [42].

APPENDIX
We outline the main idea of the proof for the asymptotic distribution of D k in Theorem 1. First, we decompose D k as D k = B 1 + B 2 + B 3 − 2B 4 as given in Section 2.3. Then we compute the mean and variance of B i , 22 ZHAO, LENG, LI AND WANG i = 1, . . . , 4 as presented in Proposition 1. This step builds on the assumption of normality of the predictors and benefits from the fact that the predictor dimension goes to infinity. Comparing the orders of the variance of B i , we find that B 2 is the leading term. We then study the asymptotic distribution of B 2 , which turns out to follow a χ 2 (1) distribution. Recall in Section 2.3, we defined K p,tl = p −1 p j=1 X tj X lj , for t, l = 1, . . . , n, l p = p j=1 λ 2 j = O(p r ) and c p = max 1≤j≤p λ j , where λ j 's are the eigenvalues of the covariance matrix Σ. Furthermore, we define a p = p j=1 λ 4 j , C 1 = E(Y t Y l K p,tl ) 2 and C 2 = E[Y 2 t ( p j=1 ρ j X tj /p) 2 ] for any t = l.