{'fraction_non_alphanumeric': 0.03317330772923668, 'fraction_numerical': 0.00883341334613538, 'mean_word_length': 5.083820093457944, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DS-GCNs: Connectome Classification Using Dynamic Spectral Graph Convolution Networks with Assistant Task Training

Functional Connectivity (FC) matrices measure the regional interactions in the brain and have been widely used in neurological brain disease classification. However, a FC matrix is neither a natural image which contains shape and texture information, nor a vector of independent features, which renders the extracting of efficient features from matrices as a challenging problem. A brain network, also named as connectome, could forma a graph structure naturally, the nodes of which are brain regions and the edges are interregional connectivity. Thus, in this study, we proposed novel graph convolutional networks (GCNs) to extract efficient disease-related features from FC matrices. Considering the time-dependent nature of brain activity, we computed dynamic FC matrices with sliding-windows and implemented a graph convolution based LSTM (long short term memory) layer to process dynamic graphs. Moreover, the demographics of patients were also used to guide the classification. However, unlike in conventional methods where personal information, i.e., gender and age were added as extra inputs, we argue that this kind of approach may not actually improve the classification performance, for such personal information given in dataset was usually balanced distributed. In this paper, we proposed to utilize the demographic information as extra outputs and to share parameters among three networks predicting subject status, gender and age, which serve as assistant tasks. We tested the performance of the proposed architecture in ADNI II dataset to classify Alzheimer's disease patients from normal controls. The classification accuracy, sensitivity and specificity reach 0.90, 0.92 and 0.89 on ADNI II dataset.


Introduction
Neurological diseases usually lead to subtle structural brain abnormality. Functional MRI, which evaluates the brain activity by measuring the blood oxygen level over time, is thus a perfect tool to investigate possible brain functional changes in many neurological disorders. Considering the brain nature of functional integration and segregation, researchers assess the correlations among neuronal activities in order to analysis the brain function. A brain network forms a graph structure naturally. It contains a set of brain regions of interest (ROIs), known as nodes, and describes their connectivity, known as edges. In the context of fMRI, these edges are derived from the statistical dependencies between each pair of brain ROIs and are summarized in a symmetric matrix called functional connectivity (FC) matrix.
Note that unlike natural images that contains shape and texture information, the spatial locality of the entries of FC do not directly corresponds to the locality of brain networks. Thus, a difficult but important challenge in FC analysis is to extract efficient disease-related features. There are two major mainstreams of methods, one is to reshape FCs into vectors of features (Plis et al., 2014;Sen et al., 2016) and the other is to treat FCs as 2D images. However, both genres are problematic. A vectorized FC will loss the spatial information as discarding the topological structure of the matrix.
In brain network, graphical algorithms can be applied on FC data without loss of information. Graph kernel, which measures the inherent in the graph structure (Shervashidze, Schweitzer, van Leeuwen, Mehlhorn, & Borgwardt, 2011;Vishwanathan, Schraudolph, Kondor, & Borgwardt, 2010), is extensively used in many graph-based algorithms. Recently, researchers also applied graph kernel for neuroimaging studies. For example, Jie et al. (Jie, Zhang, Wee, & Shen, 2014) used a graphkernel-based approach to measure directly the topological similarity between connectivity networks. However, the computational complexity of graph kernel is intensive.
Graph convolution neural networks (GCN) allow an implementation of neural networks on graph structures. Neuroimaging pattern recognition applications include node classification and graph classification. Node classification assigns a pre-defined demographic graph for all subjects accompanied by a set of features. In the work of Parisot , the feature of every individual, i.e., every graph node, was a feature vector extracted from images, while the edges were calculated from the similarities between corresponding subjects. Graph classification treats each individual as an independent graph. For example, Ktena et al. (Ktena et al., 2018) proposed a Siamese graph convolution network to learn the similarity between brain networks. However, their works neglected the dynamic nature of brain activity, which can possibly improve the performance of classifiers.
Demographic information has been proved to be useful in many clinical studies.
Researchers has reported a gender difference on cognitive functioning in brain (Halpern, 2012). Besides, the incidence of many neurological diseases correlates with the years of age (Braak & Braak, 1997;Butwicka & Gmitrowicz, 2010). Thus, gender and age information was used in many neurological disease classification studies.
However, in small datasets, which is common in medical imaging scenario, we argue that the status of subjects and demographic information of subjects are not always strongly correlated because of sampling preference. For example, a major guideline of collecting data for diagnosis is to balance the demographic distribution in both patient group and healthy control group, making gender and age weakly correlates with diagnosis. In this case, involving gender and age directly in the prediction model may not help models to learn diagnosis better.
In our recent research presented at a conference (Xing et al., 2019), we proposed a novel graph convolution recurrent network for fMRI pattern recognition. Graphs are defined with time-varying edges, i.e., the dynamic functional connections, and fixed nodes, which are characterized by the structural information retrieved from average fMRI. We demonstrate the potential of our method on connectome classification tasks.
The proposed method can improve diagnosis accuracy on fMRI through three aspects: 1) A graph classification algorithm dedicated for functional connectivity data; 2) An LSTM architecture which could further extract temporal information relevant to diagnosis; 3) Multi-task settings which utilize demographic information as extra outputs for guiding the extraction of efficient features. In this work, we further extended the aforementioned study with more methodological details, validations and discussions.

Method
We introduce below the proposed dynamic spectral GCNs with assistant task training using dynamic functional connectivity matrices. First, the dynamic connectivity matrices based on correlations of BOLD signals from sliding-windows are defined as time-varying edges. Each node of the connectivity graph reflects an anatomical ROI.
The feature on each node is defined by the volume of the corresponding ROI. Second, after defining the graph structure, a spectral graph convolution-based LSTM network is employed to extract information from the dynamic connectivity graphs. Finally, we make use of the demographic information as extra inputs, by adding two assistant networks with similar structure but different parameters, predicting gender and age.
The feature maps from assistant networks were then weighted and combined with feature maps from main network, guiding the parameter training and optimizing the results.

Graph Construction
A graph ( , ℰ) was defined by two matrices, i.e., node feature matrix and adjacency matrix. Node feature matrix denoted as ∈ × , where is the number of nodes in the graph and denotes the number of features, describes the property of every node in the graph. And adjacency matrix ∈ × represents the graph structure in a matrix form.
In our method, we obtain feature matrices from T1 images by assigning the volume of each ROI into corresponding diagonal entry. Thus, the feature matrix in our where is the brain volume of -th ROI. Dynamic adjacency matrices are computed by a sliding window over the entire time series, and can be represented as { | ∈ × , = 1,2, … , }.
Here is the overall time points of dynamic adjacency matrices.

Graph Convolution LSTM
Graph Convolution. Conventional CNN performs spatial convolution on 2D or 3D images, which is reasonable because of the natural adjacent properties of neighboring pixels/voxels. However, when the input of neural network is a graph, such convolution operations should be designed specifically. In our study, we chose spectral graph convolution (Defferrard, Bresson, & Vandergheynst, 2016;Kipf & Welling, 2017), because of 1) the difficulty to match spatial local neighborhoods for a node and 2) the complete mathematical definition of spectral graph convolution. Consider a graph adjacency matrix , the normalized graph Laplacian of is Laplacian form the Fourier bases of the graph. Thus, the Fourier transform of feature maps is ̂= . Since the convolution operation on spatial domain is the multiplication on spectral domain, the graph convolution operation is defined as * = (( )⨀( )), where represents learnable parameters of the graph convolutional kernel. In our study, we used Chebyshev polynomial to approximate . This approximation enables spectral convolution to be spatial localized, and decreases the learning and computational complexity. Using Chebyshev polynomial, equation (2) can be approximated as * = ∑ (̃) .

=0
( 3) Here, Chebyshev polynomial is ( ) = 2 −1 ( ) − −2 ( ) with 0 ( ) = 1 and 1 ( ) = . ̃= 2 − is the scaled graph Laplacian, denotes the largest eigenvalue of and is the identity matrix. is one of the learnable parameters. And this expression is localized because the -th polynomial of the graph Laplacian only depends on maximum -th nearest neighbor of the central node.
For brain networks, every brain region densely connected with others, either weakly or strongly. Thus, we only need to consider the situation where = 1. We further approximate ≈ 2, and thus equation (3) is simplified as with ̃= + and ̃= (∑̃, ) . With equation (4), for input signal ∈ × , with input features for nodes and expected output features, we could define graph convolutional filter as follow: where ∈ × is a matrix of graph convolutional parameters and ∈ R × is the output feature matrix.

GC-LSTM.
Dynamic graphs require a recurrent structure to handle temporal information. In our paper, LSTM (Long Short-Tern memory) network is used, which could avoid the long-term dependency problem by recording the cell state in every time step. By replacing matrix multiplication in conventional LSTM with graph convolution, we could obtain below mathematical formulas which graph convolutional LSTM follows: Hidden State: ℎ = ⨀ ℎ ( ).
Here, * denotes the graph convolution operator. Forget gate decides what information are going to be discarded from the cell state, while input gate decides what new information to store. Cell state is updated after forget gate and input gate, and output gate calculates the output, which is then filtered by cell state and send to next time step. Details of the GC-LSTM cell is shown in Figure 2.

Figure 2.
GC-LSTM hidden cell in detail. Each cell has two sources of input, the present ( and ) and the recent past (ℎ −1 and −1 ). Output sequence {ℎ | = 1,2, … } were then treated as feature maps for following layers.

Assistant Task Training
Gender and age provide important demographic information for disease prediction.
Conventional deep learning classifiers added gender and age as additional features into the last fully connected layer. However, when adding demographic information as input features, the status and demographic information of subjects correlates in a statistical manner. For example, in ADNI datasets, gender and age are balanced among all groups, which makes these features correlates weakly with diagnosis labeling. In this situation, adding additional demographic features may not improve the classification performance. In addition, a balanced distribution in dataset does not imply a balanced distribution in real life: gender and age do affect the incidence of many neurological diseases (Hebert et al., 1995;Nebel et al., 2018). Thus, we propose to use the demographic information as extra outputs in our networks. This strategy not only could improve the classification performance of weakly correlated demographic features, but also could guide the parameter optimizing in diagnosis task.
However, conventional multitask settings adopt a hard representation-sharing manner, by which different tasks share same parameters in the beginning several convolutional layers of the network. In fact, it is difficult to decide which layers to be shared and which layers to be split among different tasks. Thus, our networks learn a linear combination of feature maps from different tasks to determine the shared representations by itself. We adopted this architecture from cross stitch networks (Misra, Shrivastava, Gupta, & Hebert, 2016), but our networks are task-centralized, i.e., only the main network, diagnosis network, receives the weighted combination of feature maps from assistant networks.
Consider the feature maps from diagnosis network, from gender prediction network and from age prediction network in layer , the linear combination of these feature maps are learnt and send into next layer in diagnosis network, is a learnable parameter in layer , which determines the contribution of demographic assistance. The loss for these networks is the weighted combination of losses from three tasks. For diagnosis and gender prediction, cross entropy loss was used, while for age prediction, we used mean square error loss.

ℒ = ℒ + ℒ + ℒ (7) Experiments
We compared the proposed method with a number of state-of-the-art classification approaches to demonstrate the improvements brought by graph convolution LSTM and assistant task training. Pre-Processing. Pre-processions of both structural and functional images are under a standard pipeline. The pre-processing steps of data is shown in Figure 4. For structural images, we performed anterior commissure (AC) -posterior commissure (  Pre-processions of fMRI include slice time correction, motion correction, registration, spatial and temporal filtering and nuisance covariates regression.

Baseline Models
We compared the proposed method with a number of state-of-the-art classification approaches on ADNI II dataset to demonstrate the improvements brought by graph convolution LSTM and assistant task training. We included 8 comparison methods in total, which compared two types of inputs, static FCs and dynamic FCs, and are shown in Figure

Results
Classification results on ADNI II dataset is shown in TABLE 2. From the table one can observe that overall graph-based networks performed better than no-graph-based methods. In addition, dynamic connectivity outperformed static connectivity methods.
When we compared different settings of our algorithm, the performance by using assistant task training outperformed those without such a training strategy. The results indicates that dynamic connectivity with graph-based neural networks could fully exploit the information in fMRI connectivity analysis. Meanwhile, assistant task training may help improve the performance of classification.

Discussion
To further demonstrate the improvements in our model and to evaluate the assistant task training strategy, we compared the computational cost of all methods for comparison, computed the class activation map of our model and plotted the contribution from assistant tasks during training. The proposed model consume less storage and operations than other methods, and located bilateral hippocampus, right precuneus, right frontal middle cortex and left precentral cortex as class activated brain regions. Feature maps from gender and age prediction tasks have shown increasing contributions to diagnosis task through training.

Computational Cost
We compared the FLOPs (Floating Point Operations) (Hunger, 2005) and the physical size of aforementioned models. Comparing to baseline models without using demographic information, the learnable parameters are tripled in our methods with soft parameter sharing. Besides, to handling dynamic FC, we implemented LSTM structure into our network, also causing a considerable increase of parameter amount.
However, as is shown in Table 3, three times larger than baseline DS-GCN, the proposed method still consume less storage and operations than state-of-the-art methods, such as SVM, CNN and RNN. We attribute it to the computational simplicity of GCN. Consider an adjacency matrix 116×116 and a node feature matrix 116×116 .
Vectorizing both matrices as input, SVM requires at least 13456 parameters to assign a prediction. However, if we adopt graph convolution as described in equation (5), the dimension of parameters needed declines into 116×1. Graph convolution allows matrix calculation on functional connectivity matrices and thus considerably reduce the number of parameters. It is also worth noting that in GCN model, we implemented two graph convolution layers to assure a stable and acceptable performance, leading GCN model more complex than DS-GCN.

Network Explainability
Despite of the superior performance of graph convolutional models, the explainability of the model is also helpful, because graph structures cannot be classified easily by human intuition. Recently, an increasing number of researches has been proposed to study the inner working of graph convolutional networks (Baldassarre & Azizpour, 2019;Pope, Kolouri, Rostami, Martin, & Hoffmann, 2019). Gradient-guided Class Activation Mapping (Grad-CAM) is a widely used algorithm to interpret the decision making procedure of neural networks (Selvaraju et al., 2016). First, Grad-CAM calculates the gradient of (probability for class , in this case, the probability for AD) with respect to feature maps 1 output from GC-LSTM layer. Then, the importance weight , of -th node in this feature map is then , = , 1 .
The class activation map M is feature map 1 filtered by importance weight: Here, activation function ReLU makes heat map focusing on brain regions which only play positive parts in classification. The class activation value of all test subjects are averaged and shown in Figure 6 and 7.

Assistant Task Training Analysis
Synthetic Experiment. To illustrate the effectiveness of assistant task learning, we reproduced the result of a synthetic experiment from (Caruana & De Sa, 1997). Here, we used two fully connected layer with activation function to approximate ( + ) 2 .
and are uniformly chosen from range [−5,5] and are encoded into 2 10 bins.
Besides the binary codes of and , another feature ( − ) 2 is also provided.
However, ( − ) 2 weakly correlates with our target (the correlation reaches zero if and are "random" enough). This does not mean we should discard ( − ) 2 .
By using ( − ) 2 as extra outputs, our simple network could easily generate the best prediction performance by learning to model sub-features and . Figure 8 shows the network design and the result of this synthetic experiment.  (Kingma & Ba, 2015) with initial learning rate 10 −3 . For random initialization, the cross entropy loss of diagnosis and gender classification were between [0.6, 0.8], while the MSE loss of age prediction were between [0.07, 0.08]. Thus, to weight the losses from different tasks into the same scale, the weights of losses were then = = 1, = 10. As seen in Eq. 6, refers to the weight of the main tasks, refers to the contribution from gender, and refers to the contribution from age. These values reflect the influences of each factor to the final classification performance. The results confirmed our hypothesis that the training of gender classification and age prediction may help generate stable and robust network feature maps for diagnosis tasks. Alphas in the last convolutional layers did not change as dramatically as the first two layers, indicating a weaker correlation among different tasks for deeper layers.

Conclusion
In this paper, we proposed a model specially to deal with dynamic functional connectivity. The novelty of this paper can be shown in three aspects. First, we proposed a specially designed graph construction algorithm, which could utilize both functional and structure images; Second, a spectral graph convolution based recurrent network is implemented to extract both functional and spatial information; Last, our model adopts a training strategy which utilizes demographic features as extra outputs, guiding the diagnosis network to train and focus.