{'fraction_non_alphanumeric': 0.03569497025419145, 'fraction_numerical': 0.012601406165494862, 'mean_word_length': 4.903895274584929, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Matrix Autoencoder Framework to Align the Functional and Structural Connectivity Manifolds as Guided by Behavioral Phenotypes

We propose a novel matrix autoencoder to map functional connectomes from resting state fMRI (rs-fMRI) to structural connectomes from Diffusion Tensor Imaging (DTI), as guided by subject-level phenotypic measures. Our specialized autoencoder infers a low dimensional manifold embedding for the rs-fMRI correlation matrices that mimics a canonical outer-product decomposition. The embedding is simultaneously used to reconstruct DTI tractography matrices via a second manifold alignment decoder and to predict inter-subject phenotypic variability via an artificial neural network. We validate our framework on a dataset of 275 healthy individuals from the Human Connectome Project database and on a second clinical dataset consisting of 57 subjects with Autism Spectrum Disorder. We demonstrate that the model reliably recovers structural connectivity patterns across individuals, while robustly extracting predictive and interpretable brain biomarkers in a cross-validated setting. Finally, our framework outperforms several baselines at predicting behavioral phenotypes in both real-world datasets.


Introduction
The brain is increasingly viewed as an interconnected network. Two key elements of this network are the structural pathways between brain regions and the functional signaling that rides on top. This structural connectivity information can be measured via Diffusion Tensor Imaging (DTI) tractography [1,23]. Likewise, resting-state fMRI (rs-fMRI) captures inter-regional co-activation, which can be used to infer functional connectivity [25,13]. Several studies have found both direct and indirect correspondences between structural and functional connectivity [16,14]. Going a step further, structural and functional connectivity have been shown to be predictive of each other at varying scales [31,8,39]. Hence, multimodal integration of these viewpoints has become a key area of focus for characterizing neuropsychiatric disorders such as autism and ADHD [28,40].
In the clinical neuroscience realm, techniques for integrating structural and functional connectivity focus on group-wise discrimination. These works include statistical tests on edge-based features to identify significant differences in Alzheimer's disease [15], parallel ICA using structure and function to identify discriminative biomarkers of schizophrenia [36], and classical machine learning techniques to predict diagnosis [6]. While highly informative at a group level, these methods do not directly address inter-individual variability, for example by predicting finer grained patient characteristics. This divide has been partially bridged by end-to-end deep learning models. Examples include MLPs [26] for age prediction from functional connectomes and convolutional neural networks [20] for predicting cognitive and motor measures from structural connectomes. The work of [11] takes the alternate approach of combining a dictionary learning model on the functional connectomes coupled with an Artificial Neural Network (ANN) that predicts multiple clinical measures, while also preserving interpretability. Even so, these models focus exclusively on a single neuroimaging modality and do not exploit the interplay between function and structure.
Geometric learning frameworks have recently shown great promise in multimodal connectomics studies, both for conventional manifold learning [38] and in the context of Graph Convolutional Networks (GCN) [27,40]. Their primary advantage is the ability to directly incorporate and exploit the underlying data geometry. Beyond associative analyses, the work of [39,4] employ multi-GCNs combined with a Generative Adversarial Network (GAN) for the alignment problem. Particularly, [39] examines the problem of recovering structural connectomes from patient functional connectomes While this paper marks a seminal contribution to multimodal integration, the representations learned by end-toend GCNs can be hard to interpret. It can also be difficult to train GANs on modest-sized datasets [19].
We propose an end-to-end matrix autoencoder that maps rs-fMRI correlation matrices to structural connectomes obtained from DTI tractography. Inspired by recent work in Riemannian deep learning [17,9], our matrix autoencoder, estimates a low dimensional embedding from rs-fMRI correlation matrices while taking into account the geometry of the functional connectivity (FC) manifold. Our second matrix decoder uses this embedding to reconstruct patient structural connectivity (SC) matrices akin to a manifold alignment [37] between the FC and SC data spaces. For regularization, the FC embedding is also used to predict behavioral phenotypes. We demonstrate that our framework reliably traverses from function to structure and extracts meaningful brain biomarkers.  notypes (green box). Let N be the number of patients and P be the number of ROIs in our brain parcellation. We denote the rs-fMRI correlation matrix for patient n by Γ n ∈ R P ×P . A n ∈ R P ×P is the corresponding structural connectivity profile, and y n ∈ R M ×1 is a vector of M concatenated phenotypic measures.

A Matrix Auto-Encoder for Connectome Manifolds
Functional Connectivity Reconstruction: By construction, the correlation matrices Γ n belong to the manifold of symmetric positive semi-definite matrices P + P . Our matrix autoencoder estimates a latent functional embedding F n ∈ P + K using a 2D fully connected (2D FC-NN) layer [9,17]. Formally, this mapping Φ ec (·) : P + P → P + K is parametrized by weights W ∈ R P ×K and is computed as a cascade of two linear layers with tied weights: F n = Φ ec (Γ n ) = W T Γ n W. Our decoder is another 2D FC-NN that estimatesΓ n from F n via a similar transformation Φ dc (·) : P + K → P + P that shares weights with the encoder. Mathematically, our FC reconstruction loss is represented as follows: The second term of Eq. (1) encourages the columns of the brain basis W to be orthonormal. Conceptually, this specialized loss helps us learn uncorrelated patterns that explain the rs-fMRI data well while acting as an implicit regularizer.
Structural Connectivity Estimation: The structural connectivity matrices A n are derived from DTI tractography and belong to the manifold of symmetric (non PSD) matrices S P . Our alignment decoder first generates an SC embedding S n ∈ R K×K from F n via a 2D FC-NN layer Φ align (·) : P + K → P + K , followed by a second 2D FC-NN layer Φ est (·) : P + K → P + P which maps to the structural connectivity matrices. For stability our SC matrices do not have self-connections and are normalized to A n 1 = 1. Accordingly, at the output layer, we apply a 2D softmax SF(·), and then suppress the diagonal elements to generate the final outputÃ n ∈ R P ×P . Our SC estimation objective is represented as follows: where • is the element-wise Hadamard product. 1 ∈ R P ×1 is the vector of all ones, and I P is the identity matrix of dimension P . Conceptually, the loss in Eq.
(2) is akin to manifold alignment [37] between the functional and structural embeddings based on a two sided Procrustes-like objective.
Phenotypic Prediction: We map the intermediate representation X n = Γ n W ∈ R P ×K learned by the FC encoder to the phenotypes y n via a cascade of a 1D convolutional layer and an ANN. The convolutional layer F conv (·) collapses X n along its rows via a weighted sum to generate a K dimensional feature vector. This feature vector is input to a simple two layered ANN G(·) to jointly estimate the elements inŷ n . We use a Mean Squared Error (MSE) loss function: This prediction task is a secondary regularizer that encourages our matrix autoencoder to learn representations predictive of inter-subject variability.
Implementation Details: We train our framework on a joint objective that combines Eqs. (1), (2) and (3) as follows: where γ 1 and γ 2 balance the tradeoff for the SC estimation and phenotypic prediction relative to the FC reconstruction objective. We employ an ADAM optimizer [22] with learning rate 0.005 and weight decay regularization [29] (δ = 0.0005) run for a maximum of 400 epochs. Optimization parameters were fixed based on a validation set consisting of 30 additional patients from the HCP database. We use this strategy to set the the dimensionality of our autoencoder embedding at K = 15 and loss penalities to {γ 1 , γ 2 } = 10 3 , 3. Finally, we utilize a spectral initialization for the encoder-decoder weights W in Eq. (1) based on the top K eigenvectors of the average patient correlation matrixΓ n for the training set. We use a similar initialization based onĀ n for Φ est (·), and default initialization [24] for the remaining layers. Our model has a runtime of 10-12 minutes on an 8 core machine with 32GB RAM implemented in PyTorch (v1.5.1).

Baseline Methods for Phenotypic Prediction
Matrix AE without rs-fMRI decoder: We start with the architecture in Fig. 1 but omit the rs-fMRI decoder loss (L FC ) in Eq. (4). This helps us evaluate the benefit of a tied encoder-decoder model for the rs-fMRI matrices.
Matrix AE without DTI decoder: We start with the architecture in Fig. 1 but remove the DTI decoder loss (L SC ) in Eq. (4). This helps us evaluate the benefit of manifold alignment to constrain the functional embedding.
Decoupled Matrix AE + ANN: We start with the architecture in Fig. 1 but decouple the representation learning on the connectomics data from the prediction of phenotypic measures by training the models separately.
BrainNetCNN: This baseline integrates multimodal connectivity data via the BrainNetCNN [20]. We modify the original architecture, which is designed for a single modality, to have two branches, one for the rs-fMRI correlation matrices Γ n , and another for the DTI connectomes A n . The ANN is modified to output M measures of clinical severity. We set the hyperparameters according to [20] rs-fMRI Dictionary Learning + ANN: The framework in [11] uses rs-fMRI correlation matrices for the prediction of multiple clinical measures. The model combines a dictionary learning with a neural network predictor, with these two blocks optimized in an end-to-end fashion via a coupled optimization objective.

HCP Dataset and Preprocessing:
We download rs-fMRI and DTI scans for of 275 healthy individuals from the Human Connectome Project (HCP) S1200 database [12]. Rs-fMRI data was pre-processed according to the standard HCP pipeline [35], which accounts for motion and physiological confounds. We selected a 15-minute interval from the rs-fMRI scans to remain commensurate with standard rs-fMRI protocols. We used the Neurodata MR Graphs package [21] to pre-process the DTI scans and estimate fiber bundles via streamline tractography. Our phenotypic measure was the Cognitive Fluid Intelligence Score (CFIS) [10,5] adjusted for age, which is obtained via a battery of tests measuring cognitive reasoning (dynamic range: 70 − 150). We used the Automatic Anatomical Labelling (AAL) atlas with 116 ROIs as our parcellation. From here, we compute the the Pearson correlation between the the mean rs-fMRI time course in the two regions; we subtract the contribution of the top (roughly constant) eigenvector to obtain the input Γ n . The input A n are obtained by dividing the number of tracts between ROIs by the total fiber tracts across all ROI pairs.
Predicting Behavioral Phenotypes: Table 1 (and Fig. 1 in Supplementary) compares the model against the baselines when predicting CFIS in a five-fold cross validated setting. Lower Median Absolute Error (MAE) and higher Normalized Mutual Information (NMI) and correlation coefficient (R) signify improved performance. Our framework outperforms the baselines during testing, though the model of [11] comes in a close second. This suggests that the Matrix Autoencoder faithfully models subject-specific variation even in unseen patients.
Functional to Structural Association: We evaluate three aspects of our functional to structural manifold alignment. First is our ability to recover structural connectivity matrices during testing. Here, we compare two distance metrics: (1) F self is the Frobenius norm between a test example A n and the model prediction for the same exampleÂ n , and (2) F other isÂ n and other SC matrices A m , (m = n). As shown in the left of Fig. 2(a), F self is consistently smaller than F other , with statistical significance determined using the Wilcoxon rank sum test. This indicates that individual differences in SC are preserved by our framework.
In the same plot, we also benchmark the recovery performance of our framework against a baseline Matrix encoder-decoder (gray box in Fig. 1) with the SC matrices as input and output. We also compare against a linear regression between the vectorized upper diagonal FC features (input) and SC features (output) to help evaluate the benefit of our matrix decomposition. As seen, our function → structure decoding achieves similar performance as directly encoding/decoding the structural connectivity. At the same time, the linear regression baseline performs worse than both of these techniques. This suggests that the ability to directly leverage the low rank matrix structure is key to preserving individual differences during reconstruction. Second, we use t-SNE to visualize the symmetric FC and SC embeddings, F n and S n , respectively. Fig. 2  from the upper-triangle entries of the embedding. As seen, the FC and SC are clustered in two different locations within this space. Interestingly, the learned representations are non-overlapping without explicit enforcement. This suggests that the alignment decoder Φ align (·) is learning a conversion between manifolds.

(b) displays the 2D t-SNE representation computed
Third, we examine the stability of the transformation learned by the alignment decoder, i.e. the weights W align ∈ R K×K of Φ align (·). We first match the columns of W align across cross validation folds according to correspondences between the functional brain basis. For each entry of W align , we compute the coefficient of variation (C v ), i.e. the ratio of the standard deviation to the mean (in absolute value). Lower values of C v indicate smaller deviations from the mean values, i.e. better stability. Fig. 2(c) displays the log coefficient of variation log(C v ), where the cool colors indicate smaller C v . As seen, a majority of the entries of W align have low variation over the mean pattern value. Overall, our results suggest that our framework learns a stable mapping across the manifolds that explains individual patterns of structural connectivity faithfully.
Examining the FC Biomarkers: We explore the functional connectivity patterns learned by our framework by first matching the brain bases (i.e., columns of W) across the cross validation folds based on the absolute correlation coefficient. We run this experiment five times with different initializations for the ANN branch to check for consistency in the learned representation. Fig 3 displays the four most consistent bases, as projected onto the brain using the region definitions of the AAL atlas. We notice that while there is spatial overlap between the bases, the standard deviations are small, which indicates that our framework is learning stable patters in the data. Subnetwork 1 highlights regions from the default mode network, which is widely inferred within the resting state literature, and known to play a critical role in consolidating working memory [34]. Subnetworks 1, 3 and 4 highlight regions from the somatomotor network and visual cortex, together believed to be important functional biomarkers of cognitive intelligence [7]. Finally, Subnetwork 2 and 4 displays contributions from the frontoparietal network and the medial prefrontal network. These areas are believed to play a role in working memory, attention, and decision making, all of which are associated with cognitive intelligence [30].
Application to a Secondary Dataset: We evaluate our framework on a second clinical cohort of 57 children with high-functioning Autism Spectrum Disorder (ASD). Rs-fMRI and DTI scans were acquired on a Philips 3T Achieva scanner (rs-fMRI: EPI, TR/TE = 2500/30ms, res = 3.05 × 3.15 × 3mm, duration = 128 or 156 time samples; DTI: EPI, TR/TE = 6356/75ms, res = 0.8 × 0.8 × 2.2mm, b-value = 700s/mm 2 , 32 gradient directions). Rs-fMRI preprocessing includes motion correction, normalization to MNI, spatial and temporal filtering, and nuisance regression with CompCorr [3]. DTI data was preprocessed using the FDT pipeline in FSL [18], with tractography performed using the BEDPOSTx and PROBTRACKx functions in FSL [2]. We define y n using three phenotypes that characterize various impairments associated with ASD: (1) the Autism Diagnostic Observation Schedule (ADOS) [33], (2) the Social Responsiveness Scale (SRS) [33], and (3) Praxis [32]. We carry forward the same parcellation scheme and model parameters as used for the HCP dataset. Table 2 compares the multi-score prediction testing performance of ADOS, SRS, and Praxis in a five fold cross validation setting. We observe that only our framework and the model of [11] can simultaneously predict all three measures. In contrast, the other baselines achieve good testing performance on one or two of the measures (for example, No DTI decoder baseline for ADOS and SRS) but cannot generalize all three. Our supplementary document includes additional results for SC recovery and FC biomarker extraction, and scatter plots for phenotypic prediction on the ASD datatset. Overall, our experiments on both healthy (HCP) and clinical (ASD) populations suggest that our model is robust across cohorts and generalizes effectively even with modest dataset sizes.

Conclusion
We have introduced a novel matrix autoencoder to map the manifold of rs-fMRI functional connectivity to the manifold of DTI structural connectivity. Our framework is strategically designed to leverage the underlying geometry of the data spaces and robustly recover brain biomarkers that are simultaneously explanative of behavioral phenotypes. We demonstrate that our framework offers both interpretability and generalizability, even for multi-score prediction on modest sized datasets. Finally, our framework makes minimal assumptions, and can potentially find application both within and outside the medical realm.