{'fraction_non_alphanumeric': 0.039696351913546486, 'fraction_numerical': 0.014801186746888104, 'mean_word_length': 4.163371844244758, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 22, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Structure Adaptive Lasso

Lasso is of fundamental importance in high-dimensional statistics and has been routinely used to regress a response on a high-dimensional set of predictors. In many scientific applications, there exists external information that encodes the predictive power and sparsity structure of the predictors. In this article, we develop a new method, called the Structure Adaptive Lasso (SA-Lasso), to incorporate these potentially useful side information into a penalized regression. The basic idea is to translate the external information into different penalization strengths for the regression coefficients. We study the risk properties of the resulting estimator. In particular, we generalize the state evolution framework recently introduced for the analysis of the approximate message-passing algorithm to the SA-Lasso setting. We show that the finite sample risk of the SA-Lasso estimator is consistent with the theoretical risk predicted by the state evolution equation. Our theory suggests that the SA-Lasso with an informative group or covariate structure can significantly outperform the Lasso, Adaptive Lasso, and Sparse Group Lasso. This evidence is further confirmed in our numerical studies. We also demonstrate the usefulness and the superiority of our method in a real data application.


Introduction
High-dimensional data occur very frequently and are especially common in genomics studies, where one of the important scientific interests is to find genomic features that yield good predictions for the response. In this paper, we focus on the high-dimensional linear regression problem where univariate responses are observed together with a high-dimensional set of predictors. To cope with the high-dimensionality of predictors, a common approach is to restrict the complexity of the model by penalizing the regression coefficients; see, for example, [12,22,27,31]. These approaches improve prediction performance and often yield a sparse estimate that facilitates feature selection.
In the context of multiple hypothesis testing, it is possible to make use of such side information to increase the statistical power of the tests [6,14,17,18,19,28]. The inclusion of such information makes the testing procedure significantly more powerful while exactly or approximately maintaining the error rate at a target level. So it is natural to ask the question of how can one incorporate such external information flexibly and robustly in the high-dimensional regression framework.
To address this, we introduce the Structure Adaptive Lasso (SA-Lasso) to incorporate any external group and covariate structure of the predictors in a high-dimensional linear regression. The basic idea behind the SA-Lasso is to translate the external information into different penalization strengths for the regression coefficients. More precisely, at each iteration of the proposed Algorithm, the penalization strength is determined jointly by the external information as well as the current estimates of the regression coefficients. When no external information is provided, our method reduces to the (iterative) Adaptive Lasso (A-Lasso). This idea is in a similar flavor to the one in structure-adaptive multiple testing, where one relaxes the p value thresholds for hypotheses that are more likely to be non-null while tightening the thresholds for the other hypotheses so that the overall error measure can be controlled.
The group Lasso and the fused Lasso are two conventional approaches for incorporating group and order information [21,23,26]. A critical difference between the SA-Lasso and these variants of the Lasso is that the SA-Lasso imposes a "soft" constraint on the regression coefficients through varying penalization strengths as compared to the "hard" constraints imposed by the group Lasso and the fused Lasso. For example, under a group structure the SA-Lasso does not force all the regression coefficients within the same group to be simultaneously zero, which is in sharp contrast to the group Lasso. Because of this, the SA-Lasso is expected to be more robust to misspecified or less informative external information. It is a desirable feature from a practical viewpoint as the informativeness of the external covariates is often unknown to researchers.
Another novel contribution of the paper is that we introduce the approximate message-passing (AMP) algorithm and the corresponding state evolution theory to the SA-Lasso framework. The AMP algorithm was inspired by belief propagation in graphical models and has made a significant impact on compressed sensing; see, for example, [3,4,9,10,11]. Here we adopt the idea to develop an AMP algorithm for the SA-Lasso estimator to speed up its computation. We further study the asymptotic risk of the SA-Lasso estimator in the large system limit (that is, n/p n → δ ∈ (0, ∞), where n denotes the sample size and p n denotes the number of features) through the state evolution equation associated with the AMP algorithm. These results shed new light on the applicability of the AMP and the state evolution theory in the structure-adaptive framework. Our numerical study confirms the practical relevance of the theory in predicting the finite sample risk of SA-Lasso. The usefulness and the superiority of our method is demonstrated through both simulations and a real data application.
The rest of the article is organized as follows. In Section 2, we define the SA-Lasso, provide some motivation behind it, and discuss ways of incorporating different structural information. Section 3 provides the AMP algorithms for the SA-Lasso (with a single iteration) under the group and the covariate-dependent structures, which is a novel contribution to the best of our knowledge. Finally, in Section 4, we validate our claims through simulation studies and a motivating application to the chronic lymphocytic leukemia (CLL) data in molecular biology and precision medicine.

Setup
Suppose we observe n samples, denoted by (y, X), arising from a linear model where y n×1 = (y 1 , y 2 , · · · , y n ) T is a response vector, X n×pn is a design matrix and ε n×1 = (ε 1 , ε 2 , · · · , ε n ) T is a vector of random errors. We further assume that (1) holds exactly for some true parameter value β 0 of β. Throughout the article, we focus on the high dimensional regime where p n grows with n and β 0 is assumed to be sparse.

Lasso and adaptive Lasso
Under the above setup, one of the most popular methods for simultaneous variable selection and estimation is the Lasso [22]. The Lasso estimator of β, denoted byβ L n , is defined aŝ where Q L n (β) = (2n) −1 y − Xβ 2 2 + λ β 1 is the objective function and a q = ( j |a j | q ) 1/q denotes the q norm of any real vector a. A notable variant of the Lasso is the adaptive Lasso (A-Lasso) [30]. The A-Lasso estimator of β, denoted byβ AL n , is defined aŝ where Q AL n (β) = (2n) −1 y − Xβ 2 2 + λ pn j=1ŵ j |β j |. Hereŵ = (ŵ 1 ,ŵ 2 , . . . ,ŵ pn ) T is a datadependent vector of non-negative weights. The basic difference between Q L n and Q AL n is that the individual non-negative weightsŵ j 's are assigned to each β j 's in addition to the common λ. This allows differential shrinkage of the components of β, and enables the resulting estimator to achieve a consistent variable selection and to correct for the bias incurred by the Lasso estimator. When p n does not depend on n, it has been proved that the A-Lasso is an oracle estimator (in the sense of [12] and [13]) and it enjoys a near-minimax optimality [30]. For γ > 0, a recommended choice is to setŵ j = |β n,j | −γ , whereβ n is a "well-behaved" preliminary estimator of β 0 [30].

Structure adaptive Lasso
In many real applications, it is possible to have some external information on the importance of each β j 's in predicting y. Here our goal is to make use of such additional knowledge in guiding us to chooseŵ in a data-dependent fashion. In general, let us refer to such external information as the structural information. Some common examples of such a structure within the components of β include group information, a monotonic ordering of their magnitudes, graph-based information, extrinsic covariate information, and so on. In this section, we propose a method of utilizing these auxiliary information together with (y, X) in choosingŵ. In what follows, we first provide the motivation and then introduce the algorithm for obtaining the proposed estimator. We also discuss some examples of structural information and derive the data-adaptive weights in each of those cases.

Motivation
Suppose the random noise ε in (1) is Gaussian with mean 0 and covariance matrix σ 2 I n for some σ 2 > 0 (note that the Gaussian assumption is only used to motivate our procedure). Then, the likelihood of β based on the observed data (y, X) is given by To encourage sparsity, we consider a joint prior distribution on (β, w). The prior is hierarchically defined as follows: first, for all j = 1, . . . , p n and for w j > 0 we let where DE(b) denotes the double-exponential distribution with some positive scale parameter b. Second, we assume w ∈ M ⊆ [0, C U ] pn for some 0 < C U < ∞, where M encodes the structural information as is described in Section 2.3.4. In practice, we set C U to be a sufficiently large positive number (say 10 30 ). Under this constraint, we assume that the joint prior density of w is proportional to Here h is the following prior density where C 1 and C 2 are constants such that the integration of pn j=1 h(w j ; γ) over M is equal to one. Based on this hierarchical setup, the negative logarithm of the joint posterior of (β, w) (up to an addition by a constant and a multiplicative constant) is given by where λ n = σ 2 /n and g(w j ; γ) = w j h(w j ; γ). If σ is known, we interpret w in (3) as a vector of hyper-parameters and aim to estimate it (together with β) by maximizing the joint posterior density. But even in this case, λ n = σ 2 /n may not be a good choice from a theoretical point of view. So for a general purpose, we replace the term by some tuning parameter λ n > 0. Thus we treat the minimization of (3) as a frequentist approach similar to the Lasso or the A-Lasso, and estimate λ n through cross-validation. This provides a direct way of incorporating external information and makes our setup widely applicable.

Estimator and algorithm
Given λ n > 0, we define the SA-Lasso estimatorβ S n as β S n ,ŵ n = arg min where Q S n (β, w) equals to and the definition of g is the same as in (3). Since Q S n is not jointly convex, instead of directly solving (4) we propose an iterative approach in Algorithm 1.
4. Constrained optimization: Given β, solve the following constrained optimization problem 5. Iterate between Steps 3 and 4 until the T th iteration.
In this algorithm, from the 1 st iteration onward, the SA-Lasso boils down to solving the A-Lasso, and its adaptive weights are updated using the most recent estimate of β. We refer to the iteration with the initial choice of w as the 0 th iteration. Following this, the algorithm can be narrated as follows: (0) At the 0 th iteration, the SA-Lasso performs the Lasso, as all the weights are initialized at 1.
(1) At the 1 st iteration, the SA-Lasso first updates the weights by using the Lasso estimates, and then the A-Lasso is carried out with the updated weights.
(2) At the 2 nd iteration, new weights are calculated by using the A-Lasso estimates from the 1 st iteration, and then the A-Lasso is carried out with the updated weights.
We repeat the procedure until T th iteration where T (≥ 1) is prefixed. Further, λ n at 0 th iteration and (λ n , γ) at every subsequent iterations are determined through one and two-dimensional crossvalidation, respectively. As for the choice of T , our numerical results deferred to sections S3.2.3 and S3.3.2 of the supplement show that the performance of Algorithm 1 with T > 1 is almost indistinguishable from that with T = 1. So for implementing the SA-Lasso, we recommend the use of Algorithm 1 with T = 1. This is further discussed in Section 4.

Structural information
We discuss two primary types of structural information that are motivated by specific applications. As a special case, we also show that the weights simplify to the well known A-Lasso weights when no structural information are provided. For ease of notation, we suppress the dependence of n on p for the rest of the discussion.
Group structure. In micro-array experiments, different genes may be clustered into several groups along biological pathways or based on phenotype information and gene ontology. This implies that the set of predictors can be partitioned into D mutually exclusive blocks In such a case, the signals are likely to appear together in these blocks. So it seems natural to consider the following set of w: and d ∈ {1, 2, · · · , D} .
In this case, the objective function in Step 4 of Algorithm 1 is convex and the minimizer can be analytically obtained. Fix any j ∈ {1, 2, · · · , p}, d ∈ {1, 2, · · · , D}, and γ ∈ (0, 1]. If j ∈ S d , then for a given β, the j th component of the minimizer is analytically given bŷ Covariate-dependent structure. In genomics studies, there are rich covariates those are potentially informative on the importance of a predictor in explaining the response. Examples include, but are not restricted to, the sum of read counts per gene across all samples in transcriptomics studies using RNA-Seq, the minor allele frequency in genome-wide association studies (GWAS), the prevalence of the bacterial species in microbiomewide association studies (MWAS), and the average methylation level of a CpG site in epigenome-wide association studies (EWAS).
Mathematically, let u j denote the external covariate associated with the j th feature lying in some generic space U ⊆ R q . The external covariate can bear information on the predictor variable x j being a signal or not, or has to do with the strength of the regression coefficient β j . But importantly, the true nature of this relationship is not known and has to be learned from the data. To incorporate the covariate information, we define the set of w as follows: where f (·; τ 0 , τ 1 ) is a smooth non-negative valued function defined on U with parameter (τ 0 , τ 1 ), and B is a compact subset of R q+1 . In particular, we assume the following parameterization: for all j = 1, 2, · · · , p and (τ 0 , τ 1 ) ∈ B, In this case, the solution to Step 4 in Algorithm 1 becomeŝ where the minimization is over (τ 0 , τ 1 ) ∈ B and max j f (u j ; τ 0 , τ 1 ) ≤ C U . Note that, for a given β, the objective function in (8) is convex in (τ 0 , τ 1 ) when B is a convex set. So the joint estimation of (τ 0 , τ 1 ) through the minimization is tractable.
No structural information. Suppose we do not have any prior structural information on β that we can take advantage of. Thus the set of w that we are interested in is M NS = [0, C U ] p . Also in this case, the objective function in Step 4 of Algorithm 1 is convex. So for a given β and for all γ ∈ (0, 1], the j th component of the minimizer is analytically given bŷ Remark 2.1. For all the types of structural information discussed above, we ensure that the objective function in (6) is convex. So the steps 3 and 4 of Algorithm 1 become convex optimizations. Remark 2.2. The group structure can be viewed as a special case of the covariate-dependent structure where the covariate u j denotes the group index that β j belongs to.

Approximate message passing algorithm and state evolution
In this section, we provide theoretical analysis for the SA-Lasso estimator obtained from Algorithm 1. For this, we focus on a single iteration (that is, T = 1). Although the results extend to the case of multiple iterations without any additional difficulty. Being more precise, under structural information our goal is to analyze the SA-Lasso estimator where the data-adaptive weights are derived from the Lasso estimator.
First, we briefly summarize some existing theories on A-Lasso. For a fixed p, its theoretical guarantees are provided in [24,30]. [16] studied the asymptotic properties of the A-Lasso estimators in sparse high-dimensional linear regression models with a fixed design matrix. For a suitable initial estimator [16] proved that, under some conditions the A-Lasso correctly selects the true nonzero coefficients with probability converging to one. The authors also showed that the estimators have the same asymptotic distribution that they would have if the zero coefficients were known in advance. Although this is an oracle property in the sense of [12] and [13], they require a fairly strong condition on the design matrix [29]. Along similar lines, [29] has defined a two-step A-Lasso procedure for linear regression and has described general model selection properties of the second stage weighted procedure for variable selection. Surveying all of these, it seems that a common practice for providing a theoretical analysis of a regularized estimator in the high-dimensional setting is to obtain an oracle inequality that provides a high-probability upper bound to the q error of the estimator. So for proving the superiority of the SA-Lasso estimator, it is informative to show that the upper bound of the q error gets improved. However, it is often unclear how tight the upper bound is in practical applications. In this paper, we take a different route by utilizing the AMP machinery to theoretically analyze our method. This approach allows us to obtain the exact asymptotic risk, which is not obtainable in the traditional theoretical framework.
Before we proceed to the details, let us provide a brief background on the AMP algorithm. The framework is inspired by belief propagation in graphical models and has made a significant impact on compressed sensing. Compressed sensing refers to a collection of signal processing techniques that focus on reconstructing high-dimensional signals in "undersampled" settings [2]. In a nutshell, compressed sensing aims at finding solutions to under-determined linear systems. Because of the smaller sample size as compared to the number of parameters, the methods developed in the earlier stage requires nonlinear and relatively expensive reconstruction schemes. One popular class of these schemes are based on linear programming (LP) methods. In spite of the theory being elegant and promising, solving the LPs in applications are more expensive than the standard linear reconstruction schemes. To reduce the computational cost and shed new light on the theoretical performance of the LP-based schemes, [9] first proposed the AMP algorithm (as a special type of iterative thresholding algorithms) and showed that it performs equivalently to the corresponding convex optimization procedure. Under the assumption that the design matrix X consists of independent and identically distributed Gaussian entries ("iid-design" from here on), the reconstruction quality of the AMP algorithm has been proven to be identical to the LP-based methods while offering a significant decrease in computational cost [3,4,9,10,11]. To our interest, [4] proposed an AMP algorithm for analyzing the Lasso estimator. Under the iid-design, it records two important findings: (i) In the large system limit, that is as n/p → δ ∈ (0, ∞), the solution from the AMP algorithm with the number of iterations growing to infinity coincides with the Lasso estimator, (ii) in the same limit, the normalized risk of the Lasso estimator converges to a quantity determined by the fixed point of the state evolution equation. Following similar footsteps and taking advantage of the general recursion algorithm proposed in [3], we analyze Algorithm 1 with a single iteration in the following two steps: (1) We introduce an AMP algorithm for the SA-Lasso. This iterative algorithm identifies the SA-Lasso estimator as a fixed point as the number of iteration goes to infinity. (2) We derive the state evolution equation for analyzing the AMP algorithm at each iteration. We obtain the asymptotic normalized risk of the estimator by taking the large system limit at any given iteration of the AMP algorithm. By letting the number of iterations go to infinity, we derive the asymptotic standardized risk of the SA-Lasso estimator.

AMP algorithm under the group structure
In this section, we propose the AMP algorithm for the SA-Lasso under the group structure defined in Section 2.3.4. For this, we assume that the true data generating parameter β 0 has the underlying group structure as described in Section 2.3.4. Following notations therein, we present the AMP algorithm of the SA-Lasso in Algorithm 2. The successive recursion that we propose here is a generalized version of the algorithm proposed in [4]. We consider a soft thresholding rule with possibly different thresholds for different groups. This enables the algorithm to be more adaptive.
For some threshold θ ≥ 0, let us denote the soft thresholding function by η( · ; θ) : R → R which is Algorithm 2 : AMP algorithm for the SA-Lasso under the group structure 1. AMP for the Lasso: Following [4], we use the AMP algorithm to get the Lasso estimator. Initialized with β 0 L = 0, the algorithm constructs the following recursion for any t ≥ 0 until convergence: 2. Determine data-adaptive thresholds using the Lasso estimates: Let β * L denotes the limiting value of the AMP Lasso estimates β t L as t → +∞. For each of the D groups we calculate the data-adaptive weights ω = (ω 1 , · · · , ω D ) T as ω d = β * L,S d −γ for γ ∈ (0, 1] (as in Section 2.3.4). Finally, using ω at each iteration t(≥ 0) we define the vector of data-adaptive thresholds for the SA-Lasso as . AMP for the SA-Lasso: Initialized with β 0 G = 0, construct the following recursion for any t ≥ 0 until convergence: defined as x + θ if x < θ. Also, by η , we denote the derivative of η with respect to its first argument. Algorithm 2 comprises of 3 basic steps: (1) using AMP to get the Lasso estimates, (2) determining data-adaptive weights based on the Lasso estimates, and (3) using AMP to get the SA-Lasso estimates. To understand how the algorithm works, let us first focus on the Step 1. For an arbitrary sequence of thresholds {θ L t } t≥0 , the recursion (9) identifies the Lasso estimates [4]. For a converging sequence of instances (according to Definition 1 in [4]), consider the sequence of vectors {β 0 (p), ε(p)} p≥0 . Let us assume that their empirical distributions converge to the probability measures P B 0 and P W , respectively. Then the asymptotic behavior of the recursion (9) can be tracked by a one-dimensional recursion defined by the sequence {τ L t } t≥0 as where The fixed point equation (11) is defined as the state evolution for the Lasso and it characterizes the AMP algorithm [4]. At each iteration t, the algorithm constructs a vector of "effective observations" X T e t L + β t L . These observations (when aggregated over components) are distributed asymptotically as B 0 + τ L t Z, where B 0 and Z ∼ N (0, 1) are independent of each other [4]. Thus the effective observations can be thought of as a noisy version of the true signal β 0 , where each entry is corrupted by a Gaussian noise with mean 0 and standard deviation τ L t . In the algorithm, η works as a denoiser on this vector and shrinks an element to 0 if its absolute value falls within θ L t . So anything below θ L t is treated as a pure noise. In order to establish a connection between the Lasso and its AMP algorithm (9), we need a specific choice for the thresholds {θ L t } t≥0 . The above interpretation of the algorithm in terms of the effective observations provides intuition for this. Since at each iteration X T e t L + β t L and B 0 + τ L t Z have the same distribution, τ L t 2 can be interpreted as the MSE of the effective observations. As θ L t provides a demarcation between a noise and a signal, it intuitively makes sense to choose θ L t to be proportional to τ L t ; that is θ L t = α L τ L t for some proportionality constant α L > 0 [4]. In fact, for a suitable choice of α L this choice of θ L t is known to be minimax optimal [4,7,8,9]. To provide an explicit connection between the Lasso and the AMP algorithm, [4] further noted that, at each iteration t, any fixed point β t L of the AMP iteration (9) with θ L t is the Lasso solution from the regularization problem at 0 th iteration of Algorithm 1 corresponding to In the limit, as t increases to ∞, the correspondence between the two for the choice θ L t = α L τ L t is given by the function where Let β * L denotes the limiting value of the AMP Lasso estimates β t L as t → +∞. In Step 2, for γ ∈ (0, 1] we calculate the vector of data-adaptive weights the inverse-power of group-wise average of the AMP Lasso estimates obtained from Step 1. Using this, at each iteration t(≥ 0) we define the vector of data-adaptive thresholds for the SA-Lasso as t plays a similar role as θ L t in the Lasso, and ω d allows the threshold of the denoiser η to vary across groups for the SA-Lasso. This makes the AMP algorithm adaptive to the group structure. To formalize the underlying group structure, we assume that the empirical distribution of β 0j converges to a mixture distribution Step 3 we define the AMP algorithm for the SA-Lasso through the recursion (10). The state evolution associated with this is given by a fixed point equation defining the sequence In the limit, as t increases to +∞ the correspondence between them for the choice η G t = α G τ G t is given by the function where Combining all the discussions above, we can summarize the complete connection between the SA-Lasso estimates from Algorithm 1 and its AMP estimates from Algorithm 2 as follows: (1) Fix α L > 0. For this, we implement the AMP algorithm for the Lasso by running the recursion (9) until convergence. Next, we obtain τ L * from the state evolution associated with it by solving the fixed point equation (11). Let β * L and τ L * are the converged values of β t L and τ L t , respectively.
(2) We determine λ L corresponding to α L from (12). Then, the Lasso solutionβ L n is obtained at the 0 th iteration of Algorithm 1 by solving the regularization problem in Step 3 corresponding to λ L .
(3) In Step 2 of Algorithm 2, we determine the vector of data-adaptive weights ω as ω = ( Because of the correspondence (12),β L n is the same as the AMP Lasso estimate β * L . This implies the data-adaptive weights are also the same. (4) We fix a suitable α G > 0 and use (14) to determine λ S corresponding to it. Given the weight vectorŵ n (β L n ), the SA-Lasso solutionβ S n is then obtained at the 1 st iteration of Algorithm 1 by solving the regularization problem in Step 3 corresponding to λ S . Given the weight vector ω, we also run the recursion (10) to get the AMP SA-Lasso estimates β * G . The fact thatβ S n is the same as its AMP estimates β * G is guaranteed by the correspondence (14).
α L in the Lasso and α G in the SA-Lasso are parameters of their corresponding AMP algorithm. After determining ω based on the AMP Lasso estimates, the state evolution (13) provide analysis for the AMP recursion (10) at each iteration t, and in the large system limit it provides asymptotic analysis for the SA-Lasso estimates under a group structure. This aids in analyzing the behavior of the AMP Algorithm 2 through a one-dimensional recursion. A detailed study of the SA-Lasso estimator under a group structure in a location model reveals that, under some conditions the upper bound for the risk of the estimator gets improved in the SA-Lasso as compared to the Lasso (see Section S1 in the supplement for more details). In fact, the amount of reduction can be substantial depending on the specifics of the underlying structure. A more sophisticated analysis based on Algorithm 2 predicts the asymptotic risk of the AMP estimate under some mild conditions and the iid-design. Our key findings and the assumptions required to obtain them are presented in Theorem 3.1.
Theorem 3.1. Consider the model in (1) and let X(p) p≥0 be a sequence of design matrices X ∈ R n×p indexed by p with iid entries X ij ∼ N (0, n −1 ). Further assume the following conditions hold: Condition C1: p and {p d } D d=1 are such that as n ↑ ∞, n/p → δ ∈ (0, ∞) and p d /p → c d ∈ (0, 1) for all d.
Condition C2: Empirical distributions of the sequence of signals {β 0S d (p d )} p d ≥0 and the noise {ε(p)} p≥0 , respectively denoted byP β 0S d andP ε , converge weakly to probability measures P B 0d and P W for all d, where for some k Denote by σ 2 = EW 2 . Then for any pseudo-Lipschitz function ψ d : where is defined by the state evolution as in (13). (1) with the underlying group structure assumed in Theorem 3.1, the theoretical analysis of a more general version of the recursion (10) is proved and is provided in Section S2.1 of the supplement. The proof of Theorem 3.1 follows as a special case of it and is discussed in Corollary S2.1 therein.
Remark 3.2. Given suitable choices of α L and α G , by (15), the asymptotic risk of the AMP SA-Lasso estimate β t G is given by In particular under the squared error loss ψ d (a, b) = (a − b) 2 and following (13), the asymptotic squared error risk simplifies to Under a group structure, we expect the right hand side to provide a good approximation to the risk of the SA-Lasso estimator proposed in Section 2.3.3. This is further confirmed through numerical studies presented in Section 4.2.
Remark 3.3. Although we have throughout discussed the AMP algorithm with a single iteration (T = 1), from the aforementioned exact correspondence between Algorithm 1 and 2 we see that there is nothing special about T = 1. If T > 1, we can repeat Step 2 and Step 3 in Algorithm 2 until the T th iteration of Algorithm 1 and its theory similarly follows from Theorem 3.1.

AMP algorithm for the SA-Lasso under the covariate-dependent structure
In a similar fashion, we can extend the results presented in Theorem 3.1 to a covariate-dependent structure. Following (8), we define with τ = (τ 0 , τ 1 ). We also assume that the empirical (joint) distribution of (u j , β 0j ) weakly converges to a distribution (U, B 0 ) ∼ P U,B 0 . In L p if we substitute β by the AMP Lasso estimates β t L , then using some modification of the AMP theory we expect that in the large system limit Algorithm 3 : AMP algorithm for SA-Lasso under the covariate-dependent structure 1. AMP for the Lasso: Following [4], we use the AMP algorithm to get the Lasso estimator. Initialized with β 0 L = 0, the algorithm constructs the following recursion for any t ≥ 0 until convergence: 2. Determine data-adaptive thresholds using the Lasso estimates: Let β * L denotes the limiting value of the AMP Lasso estimates β t L as t → +∞. Based on this, we define the vector of data-adaptive weights ω = (ω 1 , · · · , ω p ) T as ω j = f (u j ; τ min ) for all j = 1, · · · , p where τ min is the minimizer of (17) with respect to τ for any γ ∈ (0, 1]. Finally, using ω at each iteration t(≥ 0) we define the vector of data-adaptive thresholds for the SA-Lasso as AMP for the SA-Lasso: Initialized with β 0 C = 0, construct the following recursion for any t ≥ 0 until convergence: Like Algorithm 2, Algorithm 3 also comprises of 3 basic steps: (1) using AMP to get the Lasso estimates, (2) determining data-adaptive weights based on the Lasso estimates, and (3) using AMP to get the SA-Lasso estimates. In Step 1, the recursion (18) identifies the Lasso estimates for an arbitrary sequence of thresholds {θ L t } t≥0 . The asymptotic behavior of the recursion can be similarly tracked by the state evolution for the Lasso defined in (11). To establish a connection between the Lasso and its AMP algorithm (18), we consider the same choice of thresholds θ L t = α L τ L t for some proportionality constant α L > 0. In the limit, as t increases to +∞, the correspondence between the two is given by the same function (12). Let β * L denotes the limiting value of the AMP Lasso estimates β t L as t → +∞. In Step 2, for γ ∈ (0, 1], we calculate the vector of data-adaptive weights ω = (ω 1 , · · · , ω p ) T , where ω j = f (u j ; τ min ) for all j = 1, · · · , p and τ min is the minimizer of (17) with respect to τ for any γ ∈ (0, 1]. Using this, at each iteration t(≥ 0), we define the vector of data-adaptive thresholds for the SA-Lasso as t plays a similar role as θ L t in the Lasso and η G t in Algorithm 2, and ω j allows the threshold of the denoiser η to vary across variables for the SA-Lasso. This makes the AMP algorithm adaptive to the covariate-dependent structure. Based on θ C t , in Step 3, we define the AMP algorithm for the SA-Lasso through the recursion (19). The state evolution associated with this is given by a fixed point equation defining the sequence {τ C t } t≥0 as where we can similarly interpret τ C t as the MSE of the effective observations constructed at each iteration of the AMP algorithm for the SA-Lasso. In order to make a connection between the SA-Lasso estimates from Algorithm (1) and the AMP SA-Lasso estimates from (19) To make the connection more clear, first let us assume that the weight vector ω is fixed. Then at each iteration t, any fixed point β t C of the AMP recursion (19) with θ C t is the A-Lasso estimate in Step 3 of Algorithm 1 corresponding to In the limit, as t increases to +∞ the correspondence between the two for the choice η C t = α C τ C t is given by the function where τ C * ≡ τ C * (α C ) = lim t→+∞ τ C t and Θ C * = α C τ C * f (U 0 ; τ min ). Combining all the discussions above, we can summarize the complete connection between the SA-Lasso estimates from Algorithm 1 and its AMP estimates from Algorithm 3 as follows: (1) Fix α L > 0. For this, we implement the AMP algorithm for the Lasso by running the recursion (18) until convergence. Next, we obtain τ L * from the state evolution associated with it by solving the fixed point equation (11). Let β * L and τ L * are the converged values of β t L and τ L t , respectively. (2) We determine λ L corresponding to α L from (12). Then, the Lasso solutionβ L n is obtained at the 0 th iteration of Algorithm 1 by solving the regularization problem in Step 3 corresponding to λ L .
(3) In Step 2 of Algorithm 3, we determine the vector of data-adaptive weights ω as ω = (ω 1 , · · · , ω p ) T where ω j = f (u j ; τ min ) for all j where τ min is the minimizer of (17) with respect to τ for any γ ∈ (0, 1]. Analogously, for any γ ∈ (0, 1] in Step 4 of Algorithm 1, we obtain the data-adaptive weightsŵ n (β L n ) = f u j ;τ n (β L n ) following (8). Because of the correspondence (12),β L n is the same as the AMP Lasso estimate β * L . This implies the data-adaptive weights are also the same.
(4) We fix a suitable α C > 0 and use (21) to determine λ S corresponding to it. Given the weight vectorŵ n (β L n ), the SA-Lasso solutionβ S n is then obtained at the 1 st iteration of Algorithm 1 by solving the regularization problem in Step 3 corresponding to λ S . Given the weight vector ω we also run the recursion (19) to get the AMP SA-Lasso estimates β * C . The fact thatβ S n is the same as its AMP estimates β * C is guaranteed by the correspondence (21).
α L in the Lasso and α C in the SA-Lasso are parameters of their corresponding AMP algorithms. After determining ω based on the AMP Lasso estimates, the state evolution (20) provide analysis for the AMP recursion (19) at each iteration t, and in the large system limit it provides asymptotic analysis for the SA-Lasso estimates under the covariate-dependent structure. This helps us analyze the behavior of the AMP Algorithm 3 through a one-dimensional recursion. The expectations in (20) and (21) are taken with respect to the joint distribution of (U 0 , B 0 ). Remark 3.4. Given suitable choices of α L and α C , similar to (15), the asymptotic risk of the AMP SA-Lasso estimate β t C is given by In particular under the squared error loss ψ(a, b) = (a − b) 2 and following (20), the asymptotic squared error risk simplifies to Under a covariate-dependent structure, we expect the right hand side to provide a good approximation to the risk of the SA-Lasso estimator proposed in Section 2.3.3. This is further confirmed through numerical studies presented in Section 4.2. Remark 3.5. Although we have throughout discussed the AMP algorithm with a single iteration (T = 1), from the aforementioned exact correspondence between Algorithm 1 and 2 we see that there is nothing special about T = 1. If T > 1, we can repeat Step 2 and Step 3 in Algorithm 3 until the T th iteration of Algorithm 1 and its theory similarly follows.

Numerical results
In this section, we conduct numerical studies where we assume that the true β is sparse and the external group or covariate information is highly informative. We compare the finite sample performances of the Lasso, the A-Lasso, the sparse group Lasso (SGL), the SA-Lasso with T = 1 (SA-Lasso (one iter)) and T = 10 (SA-Lasso) through simulations and a real data example.

Simulation setups
For generating β 0 , under a group structure, we assume that there are four underlying groups. In each group, we randomly generate β 0,j from a 2-groups mixture distribution of 0 and N (µ, s 2 ) for some µ and s. Of these four groups, one group primarily corresponds to the null group; that is, it generates 0 with a high probability and nonzero signals with a small probability. The other three groups primarily correspond to the signal groups; that is, it generates 0 with a small probability and nonzero signals with a high probability. Under the covariate dependent structure, we generate real-valued covariates (u 1 , · · · , u p ) T corresponding to (β 0,1 , · · · , β 0,p ) T = β 0 . To generate β 0 using this covariate information, we randomly generate β 0,j from a 2-groups mixture distribution of 0 and N (µ, s 2 ) (for some µ and s), where it is 0 with probability logit −1 (u j ). Given β 0 and a design matrix X, we then generate the response y from model (1) with error variance σ 2 . For a more detailed description of the setup, please refer to Section S3.1 in the supplement.

State evolution prediction as a finite sample approximation
In this section we compare the squared error risk of the SA-Lasso (one iter) based on finite samples with the theoretical risk predicted by the state evolution equation under both group and covariatedependent structures. For this comparison, we have set n/p = δ = 0.64, p = 500 and σ 2 = 0.2. We consider two choices for X: namely, the iid-design and the binary design. The iid-design is the same as that in Section 3, whereas in the binary design each entry independently equals to +1/ √ n or −1/ √ n with equal probability.    (13) and (20) depend on α L , the AMP parameter for the Lasso. So for deriving the asymptotic risk for the SA-Lasso we choose this to be the value for which the risk of the Lasso estimator, as predicted by δ[(τ L * ) 2 − σ 2 ] (the pink line in Figure 1), is minimized. This makes the numerical analysis comparable with the Algorithm 1. The results presented in the figure suggest that the risks predicted by the state evolutions (13) and (20) are already very close to the finite sample risk even for p = 500. This numerically justifies our theory. In fact, the prediction under the iid-design seems to match very well even for the binary design. In addition, it also shows that the minimum MSE obtained by the SA-Lasso is significantly improved (more than 50%) over the Lasso and the A-Lasso.

Performance of the SA-Lasso
In this section we investigate the performance under the iid-design (as in Section 4.2), an AR(1) design with ρ = 0.5 and an equicorrelated design with ρ = 0.5. The same simulation setup as in Section 4.2 is used with (i) p = 300, and (ii) δ is varied as 0.2, 0.4, . . . , 1. At each iteration of Algorithm 1 we use a 10-fold cross-validation to jointly determine (λ n , γ), and γ is exact up to 2 decimals. The performances are summarized over 100 replications.

Informative structural information
In this section, we investigate the gain in performance by the SA-Lasso as compared to other methods when the underlying structural information is actually informative. We have considered 3  different design matrices for both group and covariate-dependent structure, and compared the MSE for performance evaluation. In Figure 2, we compare the SA-Lasso with Lasso, A-Lasso, and SGL (under the group structure). For the SA-Lasso we consider Algorithm 1 with T = 1 and T = 10, and they are respectively referred in the figure as SA-Lasso (one iter) and SA-Lasso. Figure 2 reveals that when δ is small, which is often the case in real life, the SA-Lasso reduces the MSE by a huge amount (sometimes as large as 80%) under all the designs and both the structures, and the dominance holds almost uniformly over δ. Let us look at the performance under the group structure more closely. Under the iid-design, that is when the features are uncorrelated, the SGL performs slightly better than the Lasso and the A-Lasso, and becomes very similar to each other as δ increases. Under the AR(1) and the equicorrelated design, the SGL performs better than the Lasso and the A-Lasso for smaller δ. But as δ increases, the stronger group constraint more prominently hurts the SGL as the features are correlated to each other.

Robustness with respect to structural information
In many real life applications, it is possible to identify some underlying group structure among the covariates, and also gather some external information on them. In this research, our goal is to make use of such information for better estimation of β. In Figure 2, we show that when δ is small and the underlying structure is informative, the SA-Lasso provides a significant improvement in terms of MSE over the other methods. But in real life, the informativeness of a plausible structure is almost always unknown. So a desirable quality that we should look for among methods aiming to exploit structural information is that it should not break down when the underlying structure is not informative. To investigate robustness with respect to the informativeness, we analyzed the methods extensively by varying both the proportion of signals and heterogeneity of a structure generating the true β. The same methods and design matrices as in Section 4.3.1 are considered for this comparison. In Figures 3-8, we respectively compare the MSE in case of the iid, the AR(1) and the equicorrelated designs under both group and covariate-dependent structure. Let us first compare the SA-Lasso with the Lasso and the A-Lasso. All the figures suggest that under a weakly informative structure, all the methods perform similar to each other for a varied proportion of signals. As the structural information gets stronger, we see a significant improvement in performance of the SA-Lasso over other methods when δ is small. As δ increases their performances become similar to each other. Under the group structure, if we also compare with the SGL, all the methods perform similar to each other for weakly informatively structure. But as the structure becomes more informative, we see two different types of behavior from the SGL depending on the strength of correlation among the features. Under the iid-design the features are uncorrelated. In this case, for a varied proportion of signals and δ, the SGL performs similar to (and in some cases slightly better than) the Lasso and the A-Lasso. Under the AR(1) and the equicorrelated design where the features are correlated with each other (with a moderate and a strong correlation, respectively), the SGL performs better than the Lasso and the A-Lasso for smaller δ. But as δ increases, the SGL breaks down. We have conducted a similar performance comparison for model selection in terms of the Matthews correlation coefficient, and also studies the robustness in performance for as we vary T . We observe that the SA-Lasso establishes similar superiority over the other methods and the performance stays almost the same as we increase T . These numerical results are deferred to Sections S3.2.2 and S3.2.3 in the supplement.
Combining our findings from sections 4.3.1, 4.3.2 and Section S3.2 in the supplement, we can safely conclude that the SA-Lasso (i) is fairly robust in terms of all the variations that we have considered, (ii) performs as good as the Lasso in worst cases, and (iii) provides a minor to major improvement in performance depending on the informativeness of the underlying structure.

Real data application: Predicting drug response in leukaemia samples
Following [25], we intend to predict the response from a drug based on several molecular predictors. Nowadays, using high-throughput technologies, a large number of molecular features from different The box plots in Figure 9 corresponding to 100 such random partitions reveal that the SA-Lasso, in general, improves the predictive performance. As it appears, the SA-Lasso (i) significantly outperforms the SGL for all the doses as it reduces the RMSPE by approximately 60-75%, (ii) offers about 10-20% gain over the Lasso for the doses 40µM , 10µM and 0.625µM , and (iii) provides an approximate 10-25% improvement over the A-Lasso for all the doses.
Next, in Figure 10 we analyze the variable selection consistency for each of the 9553 predictors separately for each of the 5 methods in case of five different regressions. The inclusion probability of j th -feature is calculated as the proportion of times out of 100 random partitions the j th -feature is included in the model (that is, β j is estimated as non-zero). In Figure 10 we present the results for  Figure 10 and figures S10-S13 in the supplement, we can summarize the findings as follows: (i) All the methods, except the SGL, more frequently selects the drug responses in the model over other features. In contrast to this, the SGL never selects any of the drug responses in its model and this is probably because of the "hard" group constraint that it imposes. The method instead includes the Methylation M-values more frequently and this makes the prediction quality poor (this can be seen from the boxplots in Figure 9).
(ii) Since the SA-Lasso implements an adaptive penalization and takes advantage of the structural information in the form of imposing a "soft" constraint, inclusion of the features becomes more consistent. This is a desirable property for a method, since ideally our goal is to select the 'correct' or most important features more frequently in the model.
We have also studied robustness of the SA-Lasso with respect to T , the prefixed number of iterations in Algorithm 1. For this comparison, we have varied T from 1 through 10 and compared the MSE.
The numerical results suggest that the performance of the SA-Lasso is almost invariant with respect to T . The comparison is deferred to Section S3.3.2 in the supplement.

Conclusion
In this research, we have proposed a structure-adaptive framework for estimating the sparse regression coefficients in high-dimensional linear regression. The SA-Lasso estimator is intuitive, practical to implement and effective in real-life applications. Our framework is flexible enough to incorporate various types structural information, that can arise in many genomics applications, in a straightforward manner: examples of such structures range from something as intuitive as groups to something as general as covariate information corresponding to each coefficients. Compared to the group Lasso and the fused Lasso, we do not directly impose constraints on the regression coefficients. Instead, we use the external information together with the data to jointly determine the penalization strength for each regression coefficient. In this sense, we have translated the external information into a soft constraint on the regression coefficients compared to the hard constraints imposed by the group Lasso and fused Lasso. Therefore, our method is expected to be more robust to misspecified or non-informative external information. For general purposes, we recommend the use of the SA-Lasso with a single iteration (that is, using Algorithm 1 with T = 1). Under the iid-design when p grows in the same order as of n, we introduce an AMP algorithm to analyze the SA-Lasso estimator with one iteration. Using this we study the asymptotic risk of our estimator through the state evolution equation associated with the AMP algorithm. A numerical study confirms the practical relevance of our theory in predicting the finite sample risk of the SA-Lasso. Although the risk is obtained under an asymptotic setting, the prediction offered by the AMP theory seems to hold even when p is as small as 500. This justifies the finite sample validity of the predicted asymptotic risk, and also confirms the practical relevance of our theory in predicting the finite sample risk of the SA-Lasso estimator. In conclusion, the promising finite sample performances demonstrated via simulations and a real data illustration suggest that the framework might be useful in a variety of statistical problems.

Supplementary Materials
Supplementary materials, which are available online, contain technical details and additional materials for the main article. Section S1 provides some analyses in a location model under the group structure. This provides a theoretical motivation for the framework proposed here. In Section S2 we present a general version of Theorem 3.1 and a technical lemma required for the proof. Lastly, some additional numerical results and more details pertaining to the simulation setups are presented in Section S3.

Abstract
In this supplementary file, we provide some technical details and additional materials for the main article.

S1 Theoretical analysis in the location model under a group structure
To get some insights on how SA-Lasso can become superior by making use of the auxiliary information, we study the L 2 -risk of the estimator under the group structure in this section. For this, we particularly consider SA-Lasso with a single iteration. Consider the location model Y = µ + ε, where µ n×1 = (µ 1 , µ 2 , · · · , µ n ) T and everything else is as in section 2.1 in the main article. Also assume that µ has an underlying group structure as described in section 2.3.4 in the main article with |S d | = n d for 1 ≤ d ≤ D. For all i ∈ S d , define the thresholding estimatorμ i of µ i as follows:μ where for some properly chosen τ > 0, sgn(x) = x/|x| and (x) + = x1{x ≥ 0}. Denote by φ(x) the density function for standard normal random variable. Theorem S1.1. Consider the above setup and in addition assume that ε i 's are independent and identically distributed as N (0, σ 2 ). Further for simplicity, let µ i = 0 if i ∈ S 1 (corresponds to the null group), and (1). Under these assumptions, if then an upper bound for the L 2 -risk ofμ (as defined in (1)) is given by

, this is larger than the leading term of the upper bound in (3).
Proof of Theorem S1.1 . For all i = 1, . . . , n and d = 1, . . . , D if i ∈ S d for some d, then based on the definition (1) define Below we study the risk of the estimator in the d th group, and the result will follow from the summation over all groups. For the ease of notation, write s = n d and λ = λ d . By the Stein's lemma, we have Direct calculation shows that On the one hand, In (4), we have used the fact that λ Using the other bound, we get We divide the rest of the arguments into the following 5 steps.
Step 1: Notice that [|Y i | − τ ] + 's are sub-Gaussian, i.e., for all i for all t ∈ R and some η i > 0. By the concentration inequality for sub-Gaussian variable, we obtain Step 2: We study the term P (M d |Y i | > λ). Using the result in Step 1, we have

S2
Let g(µ i ) = P (|Y i | > λ(EM d + ξ) −1 ). As g is symmetric about zero, By the Mill's ratio, which then implies that Step 3: Next we study the term E λ Step 4: Using the above results, we have Thus we get Under assumption (2), (2φ(1) + 1)a 2 > E λ M d 2 + σ 2 for all d > 1. The above bound then becomes For d = 1, by choosing ξ = s −1/3 , we have EM 1 = M 0 and γ = M 0 /(M 0 + s −1/3 ). It is not hard to verify that s 1−γ 2 → 1. Then we have E μ S1 − µ S1 Step 5: Under condition (2) and combining the results above, we finally obtain Here φ denotes derivative with respect to the first coordinate of φ.

S3.1 State evolution prediction
In this subsection we provide the details of the simulation setups for Section 4.1 in the main article. iid ∼ π 0d δ 0 + (1 − π 0d ) N (µ 0d , 0.3 2 ) for j in S d for all d.

S3.2 Simulation study
In this section we conduct a more exhaustive analysis. As in Section 4.2 in the manuscript, we vary the proportion of signals and heterogeneity of an underlying structure in the true β to examine the robustness of the proposed method. Our primary interest is to investigate the performance under varying strength of structural information. This is mainly because in real life we are completely unaware of both the true underlying structure and the informativeness of the structural information that we intend to use. The hope is that the MSE decreases significantly when the intended structure is actually informative, and at the same time does not get remarkably worse (possibly perform as good as the LASSO or the A-Lasso) when the information is not much informative. The results that follow actually examines this.

S3.2.1 Central simulation conditions
The simulation setup is similar to section S3.1 with some modifications. For the setup description let us denote the proportion of signals in β 0 by η. The goal is to control π 0 = (π 01 , · · · , π 0D ) so that the expected proportion of signal is η. Next we describe the setup under group and covariate dependent structure separately. Covariate-dependent structure. Here also the motivation is to make a 'smooth' version of the above described group structure. For this we note that the heterogeneity in the structure can be varied in exactly the same way. But for varying the signal proportion we reparameterize π l 0 and π u 0 in terms of η. Suppose we want to vary the signal proportion between η l and η u . Consider c and p d same as in the group structure above. Then given D = 4 and an odds vector O, π l 0 and π u 0 are defined as π l 0 := π l 0 (η l ) = min 1, Following this we generate the external covariate u as u j iid ∼ U logit(π l 0d ), logit(π u 0d ) for j in S d for all d = 1, · · · , 4. These determine π 0j 's as π 0j = logit −1 (u j ). Lastly, based on these π 0 and µ, β 0j 's are generated from the same 2-groups mixture model as in section S3.1. The signal proportion is varied as follows: • Sparse signal: (η l , η u ) = (0.15, 0.2).
For each type of structural information, we also vary the design matrix X as the iid-design, an AR(1) design with ρ = 0.5 and an equicorrelated design with ρ = 0.5. The performances are summarized over 100 replications.

S3.2.2 Model selection performance
Under the simulation settings described in Section S3.2.1, we have compared the model selection performance for different methods. Here we use the Matthews correlation coefficient (MCC) to quantify the quality of zero/non-zero classifications of β j 's. By definition, the MCC is a correlation coefficient between the observed and predicted binary classifications and so it takes values between -1 and +1. A coefficient of +1 indicates a perfect classification, 0 means performance is similar to that of a random classification, and -1 indicates a total disagreement between prediction and observation. Figures S1-S3 represent the MCC performances for all the simulation settings under both group and covariate-dependent structure. The findings from the figures can be summarized as follows: (i) In the iid design, all the model selection performances are on the poor side when structure is not much informative. The performance gets improved as the heterogeneity of the underlying structure increases. The improvement in the MCC is particularly more prominent when δ is small. Except the case when an underlying group structure in weakly informative and the signal is sparse, model selection quality of the SGL becomes poor as δ increases. Contrary to this, the SA-Lasso performs either better than or as good as the other methods uniformly over δ.
(ii) In case of correlated designs, the A-Lasso performs better than others when the information is weak and δ is on the higher side. On the contrary, the SGL starts to perform poorly as δ, or proportion of signal, or the informativeness of the underlying structure, or the correlation among the predictors S7  increases. When δ is small the SA-Lasso accomplishes (slightly or significantly) better average MCC and it eventually becomes as good as the Lasso and the A-Lasso as δ increases.
(iii) The performance improvement of the SA-Lasso over the SGL is more significant for an equicorrelated design as the informativeness of the structure increases, or the signal gets more sparse, or δ decreases.

S3.2.3 Robustness across iterations
In this section we analyze the performance of the SA-Lasso as we vary the number of iterations (T ) in Algorithm 1 presented in the manuscript. We study the performance as T varies from 1 through 10. From figures S4-S9 presented above, in general we can hardly see any differences in the performances of the SA-Lasso for varied iterations. We see some differences under the equicorrelated design when the structure is highly informative and the signal is sparse. The differences in performance vanishes as the signal gets more dense. Thus combining all the figures we can safely conclude that the method is quite robust with respect to the number of iterations under both group and covariate-dependent structural information for the varieties of simulation scenarios considered here.

S3.3 Real data application: Predicting drug response in leukaemia samples
In this section we refer back to Section 4.3 in the main article. In this real data analysis, our goal is to predict the response from a drug based on several molecular predictors. The CLL data that we consider here consists of several omic measurements from 121 patients [2]. In fact there are 3 different features corresponding to 3 different omic types, and these different omic types have different scales of measurement. So instead of scaling all the feature measurements to make them comparable, we intend to penalize them accordingly in an adaptive fashion. So we intend to use a 3-groups structure representing 3 omic types.

S3.3.1 Consistency in variable selection
In this section we present the results for the five methods in case of the four regression regression problems where the goal is to predict response to the Ibrutinib drug with 10 µM , 2. doses. The findings from the figures S10-S13 are the same as it is reported at the end of Section 4.3 in the manuscript.

S3.3.2 Robustness across iterations
As in Section S3.2.3, here we analyze the robustness in RMSPE performance ( Figure 6 in the manuscript) of the SA-Lasso in the real data analysis as we vary the number of iterations (T ) in Algorithm 1 in the manuscript. Like in Section S3.2.3 we vary T from 1 through 10. In the manuscript Figure 3 compares the boxplots of the RMSPE for the Lasso, the A-Lasso, and the SGL with the SA-Lasso with T = 1 and 10. To investigate the robustness of our method across iterations, for the same data analysis we compare similar boxplots of the RMSPE for the SA-Lasso corresponding to the 10 iterations. Figure S14 presents these plots separately for each of the 5 regressions.  Figure S10: Consistency in variable selection in Ibrutinib (10 µM ) response prediction. The above plot presents consistency of each method with regard to how frequently it includes a feature in the model. This plot focuses on the regression problem of predicting response to the Ibrutinib drug with 10 µM dose.
As we can see from Figure S14, the RMSPE of the SA-Lasso hardly varies across iterations. So it is fair to conclude that the number of iterations in Algorithm 1 in the manuscript does not really have any impact on the performance. So for general purposes we recommend the use of a single iteration (that is T = 1). S12