{'fraction_non_alphanumeric': 0.03755730943957125, 'fraction_numerical': 0.012602842966901836, 'mean_word_length': 4.334822425731517, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Cooperative learning for multiview analysis

We propose a new method for supervised learning with multiple sets of features ("views"). The multiview problem is especially important in biology and medicine, where"-omics"data such as genomics, proteomics and radiomics are measured on a common set of samples. Cooperative learning combines the usual squared error loss of predictions with an"agreement"penalty to encourage the predictions from different data views to agree. By varying the weight of the agreement penalty, we get a continuum of solutions that include the well-known early and late fusion approaches. Cooperative learning chooses the degree of agreement (or fusion) in an adaptive manner, using a validation set or cross-validation to estimate test set prediction error. One version of our fitting procedure is modular, where one can choose different fitting mechanisms (e.g. lasso, random forests, boosting, neural networks) appropriate for different data views. In the setting of cooperative regularized linear regression, the method combines the lasso penalty with the agreement penalty, yielding feature sparsity. The method can be especially powerful when the different data views share some underlying relationship in their signals that can be exploited to boost the signals. We show that cooperative learning achieves higher predictive accuracy on simulated data and a real multiomics example of labor onset prediction. Leveraging aligned signals and allowing flexible fitting mechanisms for different modalities, cooperative learning offers a powerful approach to multiomics data fusion.


Introduction
With new technologies in biomedicine, we are able to generate and collect data of various modalities, including genomics, epigenomics, transcriptomics, proteomics, and metabolomics (Fig. 1A). Integrating heterogeneous features on a common set of observations provides a unique opportunity to gain a comprehensive understanding of an outcome of interest. It offers the potential for making discoveries that are hidden in data analyses of a single modality and achieving more accurate predictions of the outcome (Kristensen et al. 2014, Ritchie et al. 2015, Robinson et al. 2017, Karczewski & Snyder 2018, Ma et al. 2020, Hao et al. 2021. While "multiview data analysis" can mean different things, we use it here in the context of supervised learning, where the goal is to fuse different data views to model an outcome of interest. To give a concrete example, assume that a researcher wants to predict cancer outcomes from RNA expression and DNA methylation measurements for a set of patients. The researcher suspects that: (1) both data views potentially have prognostic value; (2) the two views share some underlying relationship with each other, as DNA methylation regulates gene expression and can repress the expression of tumor suppressor genes or promote the expression of oncogenes. Should the researcher use both data views for downstream prediction, or just use one view or the other? If using both views, how can the researcher leverage their underlying relationship in making more accurate prediction? Is there a way to strengthen the shared signals in the two data views while reducing idiosyncratic noise?  Figure 1: Framework for multiomics data fusion. (A) Advances in biotechnologies have enabled the collection of a myriad of "-omics" data ranging from genomics to proteomics measured on a common set of samples. These data capture the molecular variations of human health at multiple levels and can help us understand complex biological systems in a more comprehensive way. Fusing the data offers the potential to improve predictive accuracy of disease phenotypes and treatment response, thus enabling better diagnostics and therapeutics. However, multiview analysis of omics data presents challenges such as increased dimensionality, noise and complexity. (B) Commonly-used approaches to the problem can be broadly categorized into early and late fusion. Early fusion begins by transforming all datasets into a single representation, which is then used as the input to a supervised learning model of choice. Late fusion works by developing first-level models from individual data views and then combining the predictions by training a second-level model as the final predictor. Encompassing early and late fusion, cooperative learning combines the usual squared error loss of predictions with an agreement penalty term to encourage the predictions from different data views to align.
There are two broad categories of existing "data fusion methods" for the multiview problem ( Fig. 1B). They differ in the stage at which the "fusion" of predictors takes place, namely early fusion and late fusion. Early fusion works by transforming the multiple data views into a single representation before feeding the aggregated representation into a supervised learning model of choice (Yuan et al. 2014, Gentles et al. 2015, Perkins et al. 2018, Chaudhary et al. 2018. The simplest approach is to column-wise concatenate the M datasets X 1 , . . . , X M to obtain a combined matrix X, which is then used as the input to a supervised learning model. Another type of early fusion approach projects each high-dimensional dataset into a low-dimensional space using methods such as principal component analysis or autoencoders (Wold et al. 1987, Vincent et al. 2010. Then one combines the low-dimensional representations through aggregation and feed the aggregated matrix into a supervised learning model. Early fusion approaches have an important limitation that they do not explicitly leverage the underlying relationship across data views. Late fusion, or "integration", refers to methods where individual models are first built from the distinct data views, and then the predictions of the individual models are combined into the final predictor (Yang et al. 2010, Zhao et al. 2019, Chen, Lu, Wang, Williamson, Rodig, Lindeman & Mahmood 2020, Chabon et al. 2020, Wu et al. 2020.
In this paper, we propose a new method to multiview data analysis called cooperative learning, a supervised learning approach that fuses the different views in a systematic way. The method combines the usual squared error loss of predictions with an "agreement" penalty that encourages the predictions from different data views to align. By varying the weight of the agreement penalty, we get a continuum of solutions that include the commonly-used early and late fusion approaches. Our proposal can be especially powerful when the different data views share some underlying relationship in their signals that can be leveraged to strengthen the signals.
The rest of the paper is organized as follows. In Section 2, we introduce cooperative learning and characterize its solution. This involves the iterative algorithm for general form of cooperative learning and the explicit closedform solution for cooperative regularized linear regression. We discuss its relation with early and late fusion, as well as other existing approaches, and establish theoretical underpinnings of our approach. In Section 3, we extend cooperative learning to settings when we have more than two data views. We demonstrate in Section 4 the effectiveness of cooperative learning in simulation studies, where we compare it to several commonly-used approaches. In Section 5, we apply cooperative learning on a real multiomics example and show that it achieves higher predictive accuracy on labor onset prediction. In Section 6, we discuss how cooperative learning can be extended to generalized linear models and Cox proportional hazards models. We outline how the framework can be extended to paired features and interaction models in Section 7. The paper ends with a discussion and an appendix.
2 Cooperative learning 2.1 Cooperative learning with two data views We begin with a simple form of our proposal for the population (random variable) setting. Let X ∈ R n×px , Z ∈ R n×pz -representing two data views -and y ∈ R n be a real-valued random variable (the target). Fixing the hyperparameter ρ ≥ 0, we propose to minimize the population quantity: The first term above is the usual prediction error, while the second term is an "agreement" penalty, encouraging the predictions from different views to agree. This penalty term is related to "contrastive learning" (Chen, Kornblith, Norouzi & Hinton 2020, Khosla et al. 2020), which we discuss in more detail in Section 2.7.
The solution to (1) has fixed points: We can optimize the objective by repeatedly updating the fit for each data view in turn, holding the other view fixed. When updating a function, this approach allows us to apply the fitting method for that data view to a penalty-adjusted "partial residual". For more than two views, this generalizes easily, with details given in Section 3.
The following relationships to early and late fusion can be seen immediately: • If ρ = 0, from (1) we see that cooperative learning chooses a functional form for f X and f Z and fits them together. If these functions are additive (for example, linear) then it yields a simple form of early fusion, where we simply use the combined set of features in a supervised learning procedure.
• If ρ = 1, then from (2) we see that the solutions are the average of the marginal fits for X and Z. This is a simple form of late fusion.
We explore the relation of cooperative learning to early/late fusion in more detail in Section 2.4, in the setting of regularized linear regression.
Note that this "one-at-a-time" fitting procedure is modular, so that we can choose a fitting mechanism appropriate for each data view. Specifically: • For quantitative features like gene expression, copy number variation, or methylation: regularized regression (lasso, elastic net), a generalized additive model, boosting, random forests, or neural networks.
• For images: a convolutional neural network.
• For time series data: an auto-regressive model or a recurrent neural network.
We illustrate this on a simulated image and omics example in Section 4.2.

Cooperative regularized linear regression
We make our proposal more concrete in the setting of cooperative regularized linear regression. Consider feature matrices X ∈ R n×px , Z ∈ R n×pz , and our target y ∈ R n . We assume that the columns of X and Z have been standardized, and y has mean 0 (hence we can omit the intercept below). For a fixed value of the hyperparameter ρ ≥ 0, we want to find θ x ∈ R px and θ z ∈ R pz that minimize: where ρ is the hyperparameter that controls the relative importance of the agreement term ||(Xθ x − Zθ z )|| 2 in the objective, and P x and P z are penalty functions. Most commonly, we use 1 penalties, giving the objective function: Note that when ρ = 0, this reduces to early fusion, where we simply concatenate the columns of X and Z and apply lasso. Furthermore, in Section 2.4, we show that ρ = 1 yields a late fusion estimate.
In our experiments, we standardize the features and simply take λ x = λ z = λ. We have found that generally there is often no advantage to allowing different λ values for different views. However, for completeness, in Appendix Section A, we outline an adaptive strategy for optimizing over λ x and λ z . We call this adaptive cooperative learning in our studies.
With a common λ, the objective becomes and we can compute a regularization path of solutions indexed by λ.
Problem (5) is convex, and the solution can be computed as follows. Letting Algorithm 1 Direct algorithm for cooperative regularized regression. Input: X ∈ R n×px and Z ∈ R n×pz , the response y ∈ R n , and a grid of hyperparameter values (ρ min , . . . , ρ max ).
for ρ ← ρ min , . . . , ρ max do SetX Solve Lasso(X,ỹ, λ) over a decreasing grid of λ values. end Select the optimal value of ρ * based on the CV error and get the final fit.
then the equivalent problem to (5) is This is a form of the lasso, and can be computed, for example by the glmnet package (Friedman et al. 2010).
This new problem has 2n observations and p x + p z features.
Let Lasso(X, y, λ) denote the generic problem: We outline the direct algorithm for cooperative regularized regression in Algorithm 1.
Remark A. We note that for cross-validation (CV) to estimate λ and ρ, we do not form folds from the rows ofX, but instead form folds from the rows of X and Z and then construct the correspondingX.
This leads to elastic net fitting, in place of the lasso, in the last step of the algorithm. This option is included in our publicly available software implementation of cooperative learning.
We show here an illustrative simulation study of cooperative learning in the regression setting in Figure 2A. We will discuss more comprehensive studies in Section 4. In Figure 2A, the first and second plots correspond to the settings where the two data views X and Z are correlated, while in the third plot X and Z are uncorrelated. We see that when the data views are correlated, cooperative learning offers significant performance gains over the early and late fusion methods, by encouraging the predictions from different views to agree. When the data views are uncorrelated and only one view X contains signal as in the third plot, early and late fusion methods hurt performance as compared to the separate model fit on only X, while adaptive cooperative learning is able to perform on par with the separate model.

One-at-a-time algorithm for cooperative regularized linear regression
As an alternative, one can optimize (4) by iteratively optimizing over θ x and θ z , fixing one and optimizing over the other. The updates are as follows: Cooperative learning achieves superior prediction accuracy on a test set when the data views X and Z are correlated. The y-axis shows the mean squared error (MSE) on a test set. The methods in comparison from left to right in each panel correspond to (1) Separate X: lasso applied on the data view X only; (2) Separate Z: lasso applied on the data view Z only; (3) Early fusion: lasso applied on the concatenated data views of X and Z; (4) Late fusion: separate lasso models are fit on X and Z independently and the predictors are then combined through linear least squares; (5) Coop: cooperative learning as outlined in Algorithm 1; (6) Adap Coop: adaptive cooperative learning as outlined in Algorithm 4 (see Appendix Section A). Note that the test MSE in each panel is of a different scale because we experiment with simulating the data of different signal-to-noise ratios (SNR). We conducted each simulation experiment 10 times. (B) The number of non-zero coefficients as a function of the 1 norm of the solution with different values of the weight on the agreement penalty term ρ: the solution becomes less sparse as ρ increases.
This is analogous to the general iterative procedure in (2). It is summarized in Algorithm 2.
Algorithm 2 One-at-a-time algorithm for cooperative regularized regression. Input: X ∈ R n×px and Z ∈ R n×pz , the response y ∈ R n , and a grid of hyperparameter values (ρ min , . . . , ρ max ).

Set y
to be the solution.
end end Select the optimal value of ρ * based on the sum of the CV errors and get the final fit.
By iterating back and forth between the two lasso problems, we can find the optimal solution to (4). When both X and Z have full column rank, the problem (4) is strictly convex and each iteration decreases the overall objective value. Therefore, the one-at-a-time procedure is guaranteed to converge. In general, it can be shown to converge to some stationary point, using results such as those in Tibshirani (2017). This algorithm uses fixed values for λ x , λ z : we need to run the algorithm over a grid of such values, or use CV to choose λ x , λ z within each iteration.
With just two views, there seems to be no advantage to this approach over the direct solution given in Algorithm 1. However, for a larger number of views, there can be a computational advantage, which we will discuss in Section 3.

Relation to early/late fusion
From the objective functions (3) and (4), when the weight on the agreement term ρ is set to 0, cooperative learning (regression) reduces to a form of early fusion: we simply concatenate the columns of different views and apply lasso or another regularized regression method.
Next we discuss the relation of cooperative learning to late fusion. Let X and Z have centered columns and y centered, from (6) we obtainX Assuming X and Z have full rank, and omitting the 1 penalties, we obtain the least squares estimates If X T Z = 0 (uncorrelated features between the views), this reduces to a linear combination of the least squares estimates for each block; when ρ = 1, it is simply the average of the least squares estimates for each block. The above relation also holds when we include the 1 penalties.
This calculation suggests that restricting ρ to be in [0, 1] would be natural. However, we have found that values larger than one can sometimes yield lower prediction error (see the simulation studies in Section 4).

Sparsity of the solution
We explore how the sparsity of the solution depends on the agreement hyperparameter ρ in Figure 2B. We did 100 simulations of Gaussian data with n = 100 and p = 20 in each of two views, with all coefficients equal to 2.0. The standard deviation (SD) of the errors was chosen so that the SNR was about 2. The figure shows the number of non-zero coefficients as a function of the overall 1 of the solutions, for different values of ρ. Note that the lasso parameter λ is varying along the horizontal axis; we chose to plot against the 1 norm, a more meaningful quantity. We see that the solutions become less sparse as ρ increases, much like the behavior that one sees in the elastic net.

Theoretical analysis under the latent factor model
To understand the role of the agreement penalty from a theoretical perspective, we consider the following latent factor model. Let u = (U 1 , U 2 , . . . , U n ) be a vector of n i.i.d. random variables with U i ∼ N (0, 1), y = (y 1 , . . . , y n ), x = (X 1 , . . . , X n ), and z = (Z 1 , . . . , Z n ), with We show that the mean squared error (MSE) of the predictions from cooperative learning is a decreasing function of ρ around 0 with high probability (see details in Appendix Section D). Therefore, the agreement penalty offers an advantage in reducing MSE of the predictions under the latent factor model.

Relation to existing approaches
We have mentioned the close connection of cooperative learning to early and late fusion: setting ρ = 0 or 1 gives a version of each of these, respectively. There are many variations of late fusion, including the use of stacked generalization to combine the predictions at the last stage (Garcia-Ceja et al. 2018).
Cooperative learning is also related to collaborative regression (Gross & Tibshirani 2015). This method uses an objective function of the form With 1 penalties added, this is proposed as a method for sparse supervised canonical correlation analysis. It is different from cooperative learning in an important way: here X and Z are not fit jointly to the target. The authors state that collaborative regression is not well suited to the prediction task. We note that if b xy = b zy = b xz = 1, each ofθ x ,θ z are the one-half of the least squares (LS) estimates on X, Z respectively. Hence the overall predictionŷ is the average of the individual LS predictions. This late fusion estimate is the same as that obtained from cooperative learning with ρ = 1. In addition, a related framework based on optimizing measures of agreement between data views was also proposed in Sindhwani et al. (2005), but it is different from cooperative learning in the sense that the data views are not used jointly to model the target.
Cooperative learning also has connections with contrastive learning (Chen, Kornblith, Norouzi & Hinton 2020, Khosla et al. 2020). This method is an unsupervised learning technique first proposed for learning visual representations. Without the supervision of y, it learns representations of images by maximizing agreement between differently augmented "views" of the same data example. While both contrastive learning and cooperative learning have a term in the objective that encourages agreement between correlated views, our method combines the agreement term with the usual prediction error loss and is thus supervised.
Moreover, the iteration (2) looks much like the backfitting algorithm for generalized additive models (Hastie & Tibshirani 1990). In that setting, each of f X and f Z are typically functions of one-dimensional features X and Z, and the backfitting algorithm iterations correspond to (2) with ρ = 0. In the additive model setting, backfitting is a special case of the Gauss-Seidel algorithm (Hastie & Tibshirani 1990). In cooperative learning, each of X, Z are views with multiple features; we could use an additive model for each view, i.e. f X (X) , where i and j are column indices of X and Z, respectively. Then each of the iterations in (2) could be solved using a backfitting algorithm, leading to a nested procedure.
We next discuss the relation of cooperative learning to a recently proposed method for multiview analysis called sparse integrative discriminant analysis (SIDA) (Safo et al. 2021). This method aims to identify variables that are associated across views while also able to optimally separate data points into different classes. Specifically, it combines canonical correlation analysis and linear discriminate analysis by solving the following optimization problem. Let X k = (x 1k , . . . , x n k ,k ) T ∈ R n k ×p , x k ∈ R p be the data matrix for class k, where k = 1, . . . , K, and n k is the number of samples in class k. Then, the mean vector for class k isμ k = 1 n kμk is the combined class mean vector. Assume that we have two data views X ∈ R n×px and Z ∈ R n×pz with centered columns, we want to find where S xz ∈ R px×pz is the sample cross-covariance matrix between X and Z. Here, tr(·) is the trace function, and ρ is the parameter that controls the relative importance of the "separation" term and the "association" terms in the objective. While SIDA also considers the association across data views by choosing vectors that are associated and able to separate data points into classes, it solves the problem in a "backward" manner, that is the features are modeled as a function of the outcome. Cooperative learning, in contrast, solves the problem in a "forward" manner (Y ∼ X, Z), which is more suitable for prediction.
We also note the connection between cooperative learning (regression) with the standardized group lasso (Simon & Tibshirani 2012). This method is a variation of the group lasso (Yuan & Lin 2006), and uses as the penalty term, rather than the sum of squared two norms. It encourages group-level sparsity by eliminating entire blocks of features at a time. In the group lasso, each block is a group of features, and we do not expect each block to be predictive on its own. This is different from cooperative learning, where each feature block is a data view and we generally do not want to eliminate an entire view for prediction. In addition, the standardized group lasso does not have an agreement penalty. One could in fact add the standardized group lasso penalty (14) to the cooperative learning objective, which would allow elimination of entire data views.
3 Cooperative learning with more than two data views When we have more than two views of the data, X 1 ∈ R n×p1 , X 2 ∈ R n×p2 , . . . , X M ∈ R n×p M , the population quantity that we want to minimize becomes We can also have different weights on the agreement penalties for distinct pairs of data views, forcing some pairs to agree more than others. In addition, we can incorporate prior knowledge in determining the relative strength of the agreement penalty for each pair of data views.
As with two views, this can be optimized with an iterative algorithm that updates each f Xm (X m ) as follows: As in the two-view setup above, the fitter E(·|X m ) can be tailored to the data type of each view.
For regularized linear regression with more than two views, the objective becomes This is again a convex problem. The optimal solution can be found by forming augmented data matrices as before in (6) and (7).
then the equivalent problem to (17) becomes With M views, the augmented matrix in (18) has n+ M 2 ·n rows, which could be computationally challenging to solve. Alternatively, the optimal solutionθ 1 ,θ 2 , . . . ,θ M has fixed pointŝ This leads to an iterative algorithm, where we successively solve each subproblem, until convergence. For a large number of views, this can be a more efficient procedure than the direct approach in (19) above. We include simulation studies on cooperative learning for more than two views in Appendix Section C.

Simulation studies 4.1 Simulation studies on cooperative regularized linear regression
Here we compare cooperative learning in the regression setting with early and late fusion methods in simulations. The set up is as follows. Given values for parameters n, p x , p z , p u , s u , t x , t z , β u , σ, we generate data according to the following procedure: 1. x j ∈ R n distributed i.i.d. MVN(0, I n ) for j = 1, 2, . . . , p x . 2. z j ∈ R n distributed i.i.d. MVN(0, I n ) for j = 1, 2, . . . , p z . 3. For i = 1, 2, . . . , p u (p u corresponds to the number of latent factors, p u < p x and p u < p z ): There is sparsity in the solution since a subset of columns of X and Z are independent of the latent factors used to generate y. Data sets are simulated with different levels of correlation between the two data views X and Z, different contributions of X and Z to the signal, and different signal-to-noise ratios (SNR). We consider the settings of both small p and large p regimes, and of both low and high SNR ratios. We use 10-fold CV to select the optimal values of hyperparameters. We compare the following methods: • Separate X and separate Z: The standard lasso is applied on the separate data views of X and Z with 10-fold CV.
• Early fusion: The standard lasso is applied on the concatenated data views of X and Z with 10-fold CV. Note that this is equivalent to cooperative learning with ρ = 0.
• Late fusion: Separate lasso models are first fitted on X and Z independently with 10-fold CV, and the two resulting predictors are then combined through linear least squares for the final prediction.
• Cooperative learning (regression) and adaptive cooperative learning.
We evaluated the performance based on the mean-squared error (MSE) on a test set. We conducted each simulation experiment 10 times.
Overall, the simulation results can be summarized as follows: • Cooperative learning performs the best in terms of test MSE across the range of SNR and correlation settings. It is most helpful when the data views are correlated and both contain signal (as in Figure 3A and Figure 3B). When the correlation between data views is higher, higher values of ρ are more likely to be selected.
• When only one view contains signal and the views are not correlated (as in Figure 3C), cooperative learning is outperformed by the separate model fit on the view containing the signal, but adaptive cooperative learning is able to perform on par with the separate model, outperforming early and late fusion.
• Moreover, we also find that cooperative learning tends to yield a less sparse model, as expected from the results of Section 2.5.
We include more comprehensive results across a wider range of simulation settings in Section B in the Appendix.

Simulation studies on cooperative learning with imaging and "omics" data
Here we extend the simulation studies for cooperative learning to the setting where we have two data views of more distinct data modalities, such as imaging and omics data (e.g. transcriptomics and proteomics). We tailor the fitter suitable to each view, i.e. convolutional neural networks (CNN) for images and lasso for omics. We simulate the "omics" data (X) and the "imaging" data (Z) such that they share some common factors. These factors are also used to generate the signal in the response y. We use a factor model to generate the data, as it is a natural way to create correlations between X, Z, and y. In Appendix Section F, we outline the full details of the simulation procedure. Figure 4 shows some examples of the synthetic images generated for this study.
Our task is to use the omics and imaging data to predict if a patient has a certain disease. We use CNN for modeling the imaging data and lasso for the omics data, and optimize the objective for the general form of cooperative learning as in Algorithm 1 with the iterative "one-at-a-time" algorithm outlined in Algorithm 2.
We compare cooperative learning to the following methods: (1) Only images: a simple one-layer CNN with max pooling and rectified linear unit (ReLU) activation is applied on the imaging data only; (2) Only omics: the standard lasso is applied on the omics data only; (3) Late fusion: separate models (CNN and lasso) are first fit on the imaging and omics data, respectively, and the resulting predictors are then combined through linear least squares using a validation set. We evaluated the performance based on the misclassification error on a test Healthy Sample Disease Sample Pixel Intensity Figure 4: Generated images for "healthy" and "disease" samples. One can think of the image as an abstract form of a patient's lung, with the darker spots corresponding to the tumor sites. The intensity of the dark spots on the disease samples is generated to correlate with the omics data and the signal in the outcome. set, as well as the difference in misclassification error relative to late fusion * . We consider both low and high SNR settings † . We conducted each simulation experiment 10 times.
The results are shown in Figure 5. We find that (1) late fusion achieves a lower misclassification error on the test set than the separate models; (2) cooperative learning outperforms late fusion and achieves the lowest test error by encouraging the predictions from the two views to agree; (3) cooperative learning is especially helpful when the SNR is low, while its benefit is less pronounced when the SNR is higher. The last observation makes sense, because when the SNR is lower the marginal benefit of leveraging the other view(s) in strengthening signal becomes larger.

Real multiomics studies
We applied cooperative learning (regression) to a data set of labor onset, collected from a cohort of women who went into labor spontaneously, as described in Stelzer et al. (2021). Proteome and metabolome were measured from blood samples collected from the patients during the last 120 days of pregnancy. The goal of the analysis is to predict time to spontaneous labor using proteomics and metabolomics data.
The proteomics data contained measurements for 1,322 proteins and the metabolomics data contained measurements for 3,529 metabolites. We split the dataset of 53 patients into training and test sets of 40 and 13 patients, respectively ‡ . Both the proteomics and metabolomics measurements were screened by their variance across the subjects. We extracted the first time point for each patient from the longitudinal study and predicted the corresponding time to labor. We conducted the same set of experiments across 10 different random splits of the training and test sets.
The results are shown in Table 1. The model fit on the metabolomics data achieves lower test MSE than the one fit on the proteomics data. Early and late fusion hurt performance as compared to the model fit on only metabolomics. Cooperative learning gives performance gains over the model fit only on metabolomics, outperforming both early and late fusion and achieving the lowest MSE on the test set. * Early fusion is not applicable in this setting. † The SNR is calculated based on the logits of the probabilities used to generate the class labels. ‡ The cohort consisted of 63 patients as described in Stelzer et al. (2021), but in the public dataset we only found 53 patients with matched proteomics and metabolomics data. We examined the selected features from cooperative learning and the other methods by comparing the ranking of the features based on the magnitude of their coefficients. All methods rank sialic acid binding immunoglobulin like lectin-6 (Siglec-6), a protein highly expressed by the placenta (Brinkman- Van der Linden et al. 2007), as the most important feature for predicting labor onset. As compared to the other methods, cooperative learning boosts up the ranking of features such as plexin-B2 (PLXNB2), which is a protein expressed by the fetal membranes (Singh & Aplin 2015), and Activin-A, which is highly expressed by the placenta as well (Stelzer et al. 2021). While factors such as Siglec-6, PLXNB2 and Activin-A have previously also been discovered by Stelzer et al. (2021) for labor onset prediction, C1q was only identified by cooperative learning as one of the top ten features. C1q is an important factor involved in the complement cascade, which influences implantation and fetal development (Girardi et al. 2020), and worth further investigation for its role in predicting labor onset.

Methods
Test (2) separate metabolomics: the standard lasso is applied on the metabolomics data only; (3) early fusion: the standard lasso is applied on the concatenated data of proteomics and metabolomics data; (4) late fusion: separate lasso models are first fit on proteomics and metabolomics independently and the predictors are then combined through linear least squares; (5) cooperative learning (Algorithm 1). The average of the selected ρ values is 0.9 for cooperative learning.

Cooperative generalized linear models and Cox regression
We next describe how cooperative learning can be extended to generalized linear models (GLMs) (Nelder & Wedderburn 1972) and Cox proportional hazards models (Cox 1972).
Consider a GLM, consisting of 3 components: (1) a linear predictor: η = Xβ; (2) a link function g such that E(Y |X) = g −1 (η); (3) a variance function as a function of the mean: V = V (E(Y |X)). For cooperative GLMs, we have the linear predictor as η = Xθ x + Zθ z , and an additional agreement penalty term ρ||(Xθ x − Zθ z )|| 2 with the following objective to be minimized: where is the negative log likelihood (NLL) of the data. For Cox proportional hazards models, becomes the negative log partial likelihood of the data.
We make the usual quadratic approximation to (21), reducing the minimization problem to a weighted least squares (WLS) problem, which yields where z is the adjusted dependent variable and W is the diagonal weight matrix, both of which are functions of θ x and θ z .
This leads to an iteratively reweighted least squares (IRLS) algorithm: • Outer loop: Update the quadratic approximation using the current parameterθ x andθ z , i.e. update the working response z and the weight matrix W .
• Inner loop: LettingX solve the following problem which is equivalent to (22).

Some extensions 7.1 Paired features from different views
One can extend cooperative learning to the setting where a feature in one view is naturally paired with a feature in another view. For example, if the jth column X j of X is the gene expression for gene j, and Z k is the expression of the protein k for which gene j codes. In that setup, we would like to encourage agreement between X j θ xj and Z k θ zk . This pairing need not exist for all features, but can occur for a subset of features.
Looking back at our objective function (4) for two views in the linear case, we add to this objective a pairwise agreement penalty of the form ρ 2 j,k∈P where P is the set of indices of the paired features.
This additional penalty can be handled easily in the optimization framework. For the direct algorithm (Algorithm 1), we simply add a new row toX andỹ for each pairwise constraint, while the one-at-a-time algorithm (Algorithm 2) can be similarly modified.

Modeling interactions between views
In our general objective function (1), we can capture interactions between features in the same view, by using methods such as random forests or boosting for the learners f X and f Z . However, this framework does not allow for interactions between features in different views. Here is an objective function to facilitate such interactions: where f XZ (X, Z) is a joint function of X and Z, including for example, interactions between the features in each view.
The solution to (26) has fixed points: When ρ = 0, from (26) the solution reduces to the additive model f X (X) + f Z (Z) + f XZ (X, Z). As ρ → 1, the joint term f XY → 0 and we again get the late fusion estimate as the average of the marginal predictionŝ f X (X) andf Z (Z). To implement this in practice, we simply insert learners such as random forest or boosting for f X , f Z and f XZ . The first two use only features from X and Z, while the last uses features from both.

Discussion
In this paper, we introduce a new method called cooperative learning for supervised learning with multiple set of features, or "data views". The method encourages the predictions from different data views to align through an agreement penalty. By varying the weight of the agreement penalty in the objective, we obtain a spectrum of solutions that include the commonly-used early and late fusion methods. The method can choose the degree of agreement (or fusion) in an data-adaptive manner. Cooperative learning provides a powerful tool for multiomics data fusion by strengthening aligned signals across modalities and allowing flexible fitting mechanisms for different modalities. The effectiveness of our methodology has implications for improving diagnostics and therapeutics in an increasingly multiomic world.
Furthermore, cooperative learning could be extended to the semi-supervised setting when we have additional matched data views on samples that are unlabeled. The agreement penalty allows us to leverage the signals in the matched unlabeled samples to our advantage. In addition, when we have missing values in some data views, the agreement penalty also allows us to impute one view from the other(s). Lastly, the method can be easily extended to binary, count and survival data. An open-source R language package for cooperative learning called multiview is available on the CRAN repository.

A Adaptive cooperative learning
In this section, we outline an adaptive strategy for optimizing over λ x and λ z for different data views. We call this adaptive cooperative learning. The method incorporates the values of λ x and λ z that have been adaptively optimized by the one-at-a-time algorithm (Algorithm 3) as a penalty factor in the direct algorithm (Algorithm 4). In the two-dimensional grid of λ x and λ z , our proposed strategy works by iteratively searching along one axis of λ while fixing the other constant.
Algorithm 3 One-at-a-time algorithm for adaptive cooperative learning (regression). Input: X ∈ R n×px and Z ∈ R n×pz , the response y ∈ R n , and a fixed hyperparameter ρ ∈ R. Output:θ x andθ z from the last iteration, along with the hyperparameters λ * x and λ * z and the corresponding CV errors.
Algorithm 4 Direct algorithm for adaptive cooperative learning (regression). Input: X ∈ R n×px and Z ∈ R n×pz , the response y ∈ R n , and a grid of hyperparameter values (ρ min , . . . , ρ max ).
for ρ ← ρ min , . . . , ρ max do Run Algorithm 3 with both (X,Z) and (Z,X) with the same folds for CV. Select the one with the lower sum of the two CV errors. Get the corresponding λ * x and λ * z .
Solve Lasso(X,ỹ, λ) over a decreasing grid of λ values, with a penalty factor of (1, . . . , 1, . Note that we form folds from the rows of X and Z and then construct the correspondingX. end Select the optimal value of ρ based on the CV error and get the final fit. C Simulation studies on cooperative learning for more than two data views Here we conduct simulation studies on cooperative learning for more than two data views. Specifically, we consider the setting of three data views, and this generalizes easily to more data views. We generated Gaussian data with n = 200 and p = 300 in each of the views X 1 , X 2 and X 3 , and created correlation between them using latent factors. The response y was generated as a linear combination of the latent factors, corrupted by Gaussian noise.
C.1 Simulation procedure for more than two data views The simulation for 3 data views is set up as follows.
4. For i = 1, 2, . . . , p u (p u corresponds to the number of latent factors): We compare the following methods: (1) separate X 1 , separate X 2 , and separate X 3 : the standard lasso is applied on the separate data views of X 1 , X 2 and X 3 with 10-fold CV; (2) early fusion: the standard lasso is applied on the concatenated data views of X 1 , X 2 and X 3 with 10-fold CV (note that this is equivalent to cooperative learning with ρ = 0); (3) late fusion: separate lasso models are first fitted on X 1 , X 2 and X 3 independently with 10-fold CV, and the three resulting predictors are then combined through linear least squares for the final prediction; (4) cooperative learning (regression) and adaptive cooperative learning.
We evaluated the performance based on the mean-squared error (MSE) on a test set and conducted each simulation experiment 10 times.
C.2 Simulation results for more than two data views Figure 12 and 13 show the simulation results for 3 data views. Overall, the simulation results can be summarized as follows: • Cooperative learning performs the best in terms of test MSE across the range of SNR and correlation settings. It is most helpful when the data views are correlated and contain signal (as in Figure 12A and Figure 13A). When the correlation between data views is higher, higher values of ρ are more likely to be selected.
• When only two data views are correlated and contain signal (as in Figure 12B and Figure 13C), cooperative learning also gives performance gains by leveraging the correlation through the agreement penalty, while early fusion can be outperformed by the separate models fit on the data views containing the signal.
• When only one view contains signal and the views are not correlated (as in Figure 12C), cooperative learning is outperformed by the separate model fit on the view containing the signal, but adaptive cooperative learning is able to perform on par with the separate model, outperforming early and late fusion.
A B C Figure 12: Simulation studies on cooperative regularized linear regression for more than two data views. (A) Simulation results when X 1 , X 2 and X 3 are correlated and all contain signal (tx 1 = tx 2 = tx 3 = 2), n = 200, p = 900, SNR = 1.5. The first panel shows MSE on a test set; the second panel shows the MSE difference on the test set relative to early fusion; the third panel shows the number of features selected; the fourth panel shows the ρ values selected by CV in cooperative learning. Here "Coop" refers to cooperative learning outlined in Algorithm 1 and "Adap Coop" refers to adaptive cooperative learning outlined in Algorithm 4. (B) Simulation results when only X 1 and X 2 are correlated and contain signal (tx 1 = tx 2 = 2, tx 3 = 0), n = 200, p = 900, SNR = 1.5. (C) Simulation results when X 1 , X 2 and X 3 are uncorrelated, and only X 1 contains signal (tx 1 = 2, tx 2 = tx 3 = 0), n = 200, p = 900, SNR = 1.5.
A B C Figure 13: Simulation studies on cooperative regularized linear regression for more than two data views. (A) Simulation results when X 1 , X 2 and X 3 are correlated and all contain signal (tx 1 = tx 2 = tx 3 = 2), n = 200, p = 900, SNR = 2.5. The first panel shows MSE on a test set; the second panel shows the MSE difference on the test set relative to early fusion; the third panel shows the number of features selected; the fourth panel shows the ρ values selected by CV in cooperative learning. Here "Coop" refers to cooperative learning outlined in Algorithm 1 and "Adap Coop" refers to adaptive cooperative learning outlined in Algorithm 4. (B) Simulation results when X 1 , X 2 and X 3 are correlated and all contain signal (tx 1 = tx 2 = tx 3 = 2), n = 200, p = 900, SNR = 0.6. (c) Simulation results when only X 1 and X 3 are correlated; X 1 contains more signal than X 3 , X 2 does not contain signal (tx 1 = 2, tx 2 = 0, tx 3 = 1.5), n = 200, p = 900, SNR = 1.0.
In this section, we study the mean squared error (MSE) of the cooperative learning algorithm. More precisely, letθ Let U new , X new , Y new , Z new be some new random variables generated from (28) independently of the previous data, i.e., where U new ∼ N (0, 1), ε y new ∼ N 0, σ 2 y , ε x new ∼ N 0, σ 2 x , ε z new ∼ N 0, σ 2 z independently. We focus on the MSE conditioning on x and z: The proposition establishes that the MSE is a decreasing function of ρ around 0 with high probability. In other words, the agreement penalty is helpful in reducing the mean squared error. To further interpret the above results, we study the ratio of the derivative to the MSE itself.
Proposition 3. The ratio of the derivative of MSE(x, z; ρ) at ρ = 0 and MSE(x, z; 0) satisfies d dρ [MSE(x, z; ρ) Here the notation O p (·) is used with the same meaning as in Proposition 2.
Proposition 3 presents a simple form of the ratio of the derivative to the MSE itself. The ratio quantifies by what percentage the "agreement" penalty decreases the MSE. It can been seen from this representation that this ratio depends on the structure of the factor model, and that the agreement penalty is more helpful when the sample size n is smaller. In the extreme case, when we have infinite data, i.e., when n = ∞, the derivative of the MSE is 0; in this case, we learn all the signals from the data even without the agreement penalty.

D.1 Proof of Proposition 1
Here we present a lemma that is used in the proof of Proposition 1.
Evaluating at ρ = 0, we get With this lemma, we are ready to prove the proposition. We start with writing down an explicit expression for the estimatorθ. LetX Then (29) implies thatθ = (θ x ,θ z ) T takes the following form: In particular,X andX Tỹ = (x T y, z T y) T . Therefore, We then move on to analyze the conditional distribution of u and y on x and z. By (28), we can write down a joint distribution of (U i , X i , Z i ):   then the above shows that we can express Y i as We can then use (48) -(50) to study the bias and variance of the estimatorsθ x andθ z . We start with the bias. By (52), we have E [y | x, z] = θ x x + θ z z. Therefore, Note that MSE(x, z; 0) = V (x, z; 0) + σ 2 = σ 2 + O p 1 n .
Thus, together with the result in Proposition 2, we have E Distribution of predicted versus true time to delivery for the labor onset prediction example We show in Figure 14 the distribution of predicted and true time to delivery for each patient, which gives a better sense of the quality of the predictions for the regression task. The left plot shows the distribution of time to delivery for all patients at their first time points in the longitudinal study; the right plot shows the predicted versus true time to delivery for the training and test samples. This is based on one random split of the training and test sets of 40 and 13 patients, respectively.

Distribution of Time to Delivery
Time to Delivery ( F Procedure for generating the imaging and "omics" data Here we outline the detailed procedure for data generation in the simulation study with imaging and "omics" data in Algorithm 5. The "omics" data (X), imaging data (Z), and the response y are generated such that there are correlations between X, Z, and y.