{'fraction_non_alphanumeric': 0.022016479135288383, 'fraction_numerical': 0.005581642597678745, 'mean_word_length': 4.964332892998679, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Unsupervised Clustering of Multivariate Time Series Microarray Experiments based on Incremental Non-Gaussian Analysis

Multiple expression levels of genes obtained using time series microarray experiments have been exploited effectively to enhance understanding of a wide range of biological phenomena. However, the unique nature of microarray data is usually in the form of large matrices of expression genes with high dimensions. Among the huge number of genes presented in microarrays, only a small number of genes are expected to be effective for performing a certain task. Hence, discounting the majority of unaffected genes is the crucial goal of gene selection to improve accuracy for disease diagnosis. In this paper, a non-Gaussian weight matrix obtained from an incremental model is proposed to extract useful features of multivariate time series microarrays. The proposed method can automatically identify a small number of significant features via discovering hidden variables from a huge number of features. An unsupervised hierarchical clustering representative is then taken to evaluate the effectiveness of the proposed methodology. The proposed method achieves promising results based on predictive accuracy of clustering compared to existing methods of analysis. Furthermore, the proposed method offers a robust approach with low memory and computation costs.


INTRODUCTION
Time series microarray experiments have become an indispensable technology, which allows for the monitoring of expression levels of thousands genes under a variety of conditions.
A microarray is typically a glass slide on to which DNA molecules are fixed in an orderly manner at specific locations called features. A microarray may contain thousands of features and each feature may contain a few million copies of identical DNA molecules that uniquely correspond to a gene. Therefore, expressions of two individuals are rarely the same. Genetic variation poses more challenges to analysis of microarray data. In addition, while the microarray experiment is carried out, the data tends to be noisy as a result of tissue collection and amplification of messenger RNA (mRNA) to hybridization onto the chip [2]. Time series microarray experiments are being popularly used to characterize the dynamic biological processes of genes across time. Its ability to analyze a huge number of genes in one experiment has encouraged biologists to collect more samples in microarray data [3]. Therefore, it is expected that more patients' data may become available in the future. With advances in modern technology, more genes may also be discovered by biologists. Hence, both the number of samples and genes are expected to grow and thus, the size of microarray data will increase. In this case, an appropriate method should be proposed to handle the problem of huge data dimensions by retaining the important information and eliminating noise.
Since, among the huge number of genes presented in a microarray, only a small number of genes are expected to be effective for performing a certain task. For delivering precise, reliable and interpretable results, it is desirable to identify a small subset of genes in developing gene expressions. Therefore, gene selection is the main goal to discover a reduced set of the most relevant genes. For multivariate microarray analysis, several techniques have been developed, because it is important to seek results that take into account the relationships between multiple variables, as well as within the variables [5]. Multivariate analysis is widely used to extract features and reduce the dimension of microarray datasets.
The most common multivariate analyses in dimensionality reduction are known as Principal Component Analysis (PCA) and Independent Component Analysis (ICA). These are used to summarize the time series microarray data. ICA is a method in multivariate statistical analysis used to separate data into underlying informational components [3], [6]. ICA is essentially a useful method to reveal the driving forces that underlie a set of observed phenomena. These phenomena include the firing of a set of neurons from microarray datasets. ICA has been applied to many goals, such as separation of artifacts in magnetoencephalogram (MEG) data, as well as visualization, localization, and feature extraction of the electroencephalogram (EEG) signal [7][8][9][10].
PCA projects the data to a new space in a lower dimension to capture the data with the highest variance spanned by the orthogonal principle components. This means that PCA finds a set of signals with a much weaker property than independence, while ICA finds a set of independent source signals [6], [22]. Specifically, PCA finds a set of signals that are uncorrelated to each other. In some cases, if the data are Gaussian, estimation of the model requires an orthogonal transformation [11]. However, PCA suffers from its orthogonality requirement for real world data whose distribution is not Gaussian. In probability theory, the Central Limit Theorem (CLT) states conditions under which the sum of a sufficiently large number of independent random variables, each with finite mean and variance, will be approximately normally distributed [12]. That is, mixtures of several sources tend to be more Gaussian than the distribution of the original sources [13]. PCA does provide a set of independent components but only if those components are Gaussian. Conversely, ICA is considered as a non-Gaussian factor analysis, in which ICA decomposes the statistical independent components. Many studies have shown that ICA outperforms PCA in microarray analysis.
Many applications dealing with massive microarray datasets are emerging. It is necessary to analyze the data as soon as the data arrives [7], [14]- [16]. Unfortunately, the traditional way of processing microarray datasets always treats these data as static. Besides, batch processing, especially in time series data requires time that depends on the duration t, which grows to infinity. Both classical PCA and ICA involve the calculation of Singular Value decomposition (SVD), which consumes huge amounts of memory [17]. As the space requirement also depends on t, the consumption of space is proportional to the duration t. Thus, batch mode processing always suffers from the large memory requirement and is time consuming, especially when the size of the data increases. On the fly processing is desirable to efficiently process the data. The incremental learning model is therefore proposed as a better alternative to process the data with less memory and time consumption. The incremental model works by processing the data at each input vector while the historical data are stored in a few variables. The previous values of the variables are updated by the next input vector [17]- [21]. This process is repeated until the end of the input data. Since historical values are kept in the variables, the incremental model has only a small memory requirement and thus accelerates the processing speed.
In this paper, we propose a method that integrates ideas based on an incremental approach and ICA. The proposed method computes an orthogonal weight by updating each weight vector in a predefined energy range. Upon obtaining the orthogonal weight, it is converged to become a non-Gaussian weight. This methodology works incrementally by updating the non-Gaussian weight using past variables instead of recomputing the entire dataset when new data input arrives; thus, it has a low computation cost. Our proposed method adapts the concepts of converging statistically independent components of multivariate data in an incremental way. The traditional batch method is limited by the requirement of re-computation of the entire data matrix when there is new input data. In contrast, our proposed method can efficiently integrate newly arriving data with past variables without involving the entire data matrix.
The remainder of this paper is organized as follows. Section 2 reviews the principal of ICA and our proposed methodology. The experimental results are presented in Section 3. Finally, conclusions and future work are outlined in Section 4.

PROPOSED METHOD
In this chapter, the fundamental concept of ICA is first presented. Then, a newly proposed method is modeled in order to enhance the way in which the idea of ICA is integrated incrementally to successful analysis of multivariate microarray time series data.

Fundamentals of ICA Model for Microarrays
The data from microarray experiments can be formed by large matrices of expression levels of genes (n columns) under different conditions (m rows). Time series microarray experiments can be represented in the ICA model by the following equation:  ,..., , 13 , 12 11 are some parameters representing the controlling factors.
are the biological processes that form the evolving base of the experiments. This can be summarized in that x is the sum of the amount of controlling factors, A weighted by the biological process, s. We denote the weight, W as the inverse matrix of mixing parameter, A and thus the hidden variables, s can be found using the expression. ICA finds A such that the source signals, s or components are statistically independent. After estimating the matrix A , we can compute its inverse, W to obtain the independent components as shown below: The model can be also written as: Assumptions on the ICA model are that the components s is statistically independent and that the independent components must have non-Gaussian distribution. Consider y, a linear combination of i s . We seek one of the independent components, y: By maximizing the non-Gaussianity of x w T , we can obtain the independent component y.

Incremental Non-Gaussian Analysis for Microarrays
Incremental non-Gaussian analysis for microarray expression, which integrates an incremental model with the concept of ICA to update the non-Gaussian weight, is presented here. Table 1 shows the description of the symbols used in the proposed method. The propose method will be described as follows. In the time series microarray data, n t x Â Î is the n genes measurement column-vector of each experimental sample time, t that might grow continuously to infinity. In the first experiment, the basis vector is adopted by weight vector, i w . Each of the weight vectors i w is projected onto the input vector, t x in the linear transformation of the data stream to obtain the hidden variables or components, Next, we estimate the energy, The exponential forgetting factor, λ is introduced so that new data can adapt to previous behavior in the data stream. The value of the exponential forgetting factor is between 0 and 1. The introduction of the λ value helps to reduce the huge memory usage, because there is no buffer space requirement for the whole data stream. The common choices of the exponential forgetting factor are values between 0.96 and 0.98. The exponential forgetting factor should be set to a high value, so that the data can adapt to the past values [18]. As long as the value does not vary too much, the result is similar. The magnitude of the estimates should also consider the past data captured by the participation weight vector, i w . For this reason, the update is inversely proportional to the current energy,  To make sure that there are sufficient components to represent the data, energy thresholding is applied to determine how many hidden variables are needed. The energy retained by the hidden variables, hv E is compared with the upper, E F E and lower bound energy, E f E of the original input data. If the hidden variables maintain too little energy, the number of hidden variables are increased, k. Conversely, if the maintained energy is too high, the number of hidden variables, k, will be decreased. This ensures that the energy of the hidden variables is always within the predefined specified interval of low and high energy values. Whenever a new datum arrives, the process of updating the weight vector will be repeated and the number of hidden variables will be adjusted to retain the energy of the components between the predefined low and high energy bounds. The algorithm is shown in Figure 1 below.

Datasets
To evaluate the effectiveness of our proposed method in the microarray dataset, the Gene Expression Omnibus (GEO datasets deposited by Blalock are utilized for the experiment [19]. The datasets are studied to analyze the hippocampal gene expression in the control and Alzheimer's Disease (AD) of varying severity on 31 dedicated microarrays. The samples are obtained from the Brain Bank of the Alzheimer's Disease Research Center at the University of Kentucky. Human GeneChips (HG-U133A) and Microarray Suite 5 are used for data collection. The samples with significant noise are eliminated, leaving eight control and five severe AD samples.
The unregulated genes in microarray data often contain little information, thus these unregulated genes are removed from the experiment. Therefore, we have 13 samples and 3617 genes for each sample.

Clustering of Microarray Datasets
Our proposed method decomposes the non-Gaussian weight vectors that are statistically independent. The underlying biological processes of the microarray gene expression data are more super-Gaussian than the mixture of the original sources. Only a small number of genes are expected to be changed at each pathological transition [8]. This leaves the majority of genes unaffected. Therefore, it forms a super-Gaussian distribution. Non-Gaussian analysis is thus suitable to analyze the microarray gene expression data. We perform unsupervised hierarchical clustering on the AD microarray dataset. Let the microarray data, X (m,n) be a two-dimensional m by n matrix. Each row, m contains the observation of gene profiles and each column, n shows the genes across the experiments. The data are first normalized to zero-mean and unit variance to standardize the data. This is done by subtracting the data from the mean value and then dividing the value by the standard deviation. Prior to the clustering process, the input vector and gene profiles of each experiment are decomposed into non-Gaussian weight vectors. Then, the non-Gaussian weight vectors are updated at the next input vector (gene profiles at the next experiment). This process is repeated by updating the weight of the gene profiles at each experimental observation until the last observation. Upon completion of the pre-processing on the microarray data, the final non-Gaussian weight vectors are obtained. In this experiment, the forgetting factor is determined to be 0.96 and the energy range from 95 per cent to 98 per cent. From the pre-processing process, the final dimension of the microarray data matrix is reduced to four. The data matrix used for clustering is a 13 by four matrix. Figure 2a shows the clustering result of the projection of the microarray data on the non-Gaussian weight vectors. The control and severe AD samples can be clearly discriminated using a small number of features.
Incremental PCA [17] is adopted to compare other methods in this section. The main difference between our proposed method and incremental PCA is that the decomposed components are not independent. That is, incremental PCA is similar to PCA, except that it works in an incremental fashion. The experiment is conducted in the same way for our proposed method and with the same parameter setting. Figure 2b shows the clustering result. It is evident that one AD sample, AD2 cannot be clustered correctly.
The experimental results are compared to the ICA result by Kong [16]. Figure 2c illustrates the clustering result by ICA. The entire data matrix is conducted by fastICA algorithm [8]. FastICA is repeated 50 times to alleviate the instability of the slightly different results generated from each looping. Eleven ICA latent variables are identified to sufficiently capture the significant underlying biologically information from the original data matrix. Both the control and AD severe samples can be discriminated into correct clusters. However, comparing this with our proposed method, the proposed method achieves promising results with fewer components.
PCA is a linear projection in the sense that the variables of the projection space are linear combinations of the gene expressions. In the PCA case, the result is also obtained from Kong [16]. The experiment is carried out by decomposing the gene expression matrix into principal components by preserving 95.5 per cent of the variance. The principal International Journal of Contents, Vol.8, No.1, Mar 2012 components with low variance that contained noise are removed from the clustering process. Figure 2d demonstrates the clustering result of PCA. The control samples are successfully clustered, but in the case of AD samples the AD2 sample is clustered incorrectly. We examine the abilities of the methods above in discriminating the AD samples from the control samples after the reconstruction. The reconstructed data are obtained by projecting the raw data into the latent variables found in these methods. A comparison is made against the hierarchical clustering results performed on the normalized raw data and the reconstructed data by the methods above. Figure 3a displays the clustering result of the normalized raw data. Some of the AD samples are clustered together, but the hierarchy of the cluster does not discriminate the two different clusters successfully. For our proposed method, the data are reconstructed by projecting the raw data into the four dimensional non-Gaussian weight vectors obtained by incrementally updating the weight vectors at each observation using 0.96 as the exponential forgetting factor and an energy range of 95 percent to 98 percent. Figure 3b shows the clustering result applied to the reconstructed data by our proposed method. It is evident that the control and AD samples are separated into different groups. The proposed method can improve the discriminative ability of the clustering result. Figure 3c depicts the clustering result of the reconstructed data by incremental PCA. The data is reconstructed by projecting the raw data into the orthogonal weight vectors obtained from incremental PCA. The parameter setting is the same as for our proposed method, to achieve fair comparison. The result demonstrates that incremental PCA fails to separate one AD sample, AD2, from the control samples.
In the PCA and ICA method, ten principal components captured 95.5 per cent of the variance and eleven independent components, which are identified as being involved in biological processes, are selected to reconstruct the data respectively. One AD sample, AD2, is not clustered into the AD group correctly by PCA and ICA [16]. Although more components are used for reconstruction, the proposed method shows a more promising clustering result with fewer components than those of other methods.

Qualitative Evaluation
This section compares the qualitative performance of our proposed method with that of ICA. ICA is chosen for the comparison with respect to our proposed method because both algorithms decompose non-Gaussian components. In this section, experiments are conducted on the AD microarray dataset because they have different behaviors in multivariate data. Synthetic data are augmented on these datasets so that more features and more observations are generated. Figure 4 shows the plot of execution time against number of genes on the microarray dataset. The proposed method is plotted with star symbols, whereas ICA is plotted with plus signs. In the microarray experiments, the number of observations and the other parameters setting are fixed. However, the number of genes increases in each loop, so that the execution time of different number of genes in both our proposed method and ICA can be recorded. The exponential forgetting factor is set at 0.96, the energy range is from 95 to 98 per cent and there are initially three hidden variables. When the number of genes increases, the execution times of both methods proportionally increase. However, from the graph, it is evident that ICA requires more computation time than our proposed method when the number of genes increases. As shown in Fig. 4, the proposed method only involves floating operation, whereas ICA involves covariance matrix calculation. ICA requires a longer computation time when the size of the matrix increases. Thus, our proposed method is proven more efficient when the number of genes or sources is growing.  However, when the number of experiments increases, the plot for ICA shows a dramatic upwards trend. Further, the execution time of ICA is also higher than that of our proposed method.
The graph in Fig. 5 shows that ICA takes a long time to execute. This can be explained by ICA's involving a covariance matrix calculation, requiring high levels of memory and computation time. In addition, when a new experiment is available, ICA requires re-computation of the entire microarray matrix. Therefore, more computation is required. It is proven that our proposed approach is robust in both the number of features (gene) and number of observations. Hence, it is suitable for microarray analysis.

CONCLUSION AND FUTURE WORK
It has been shown that an incremental non-Gaussian model can be used to effectively cluster data for microarray expression matrices. In the microarray dataset, the super-Gaussian weight matrix reveals the underlying biological processes in the microarray data. The clustering accuracy for the microarray shows promising results using our proposed method when compared with previous analysis such as ICA or PCA.
Furthermore, the proposed method demonstrates a robust approach, with low memory and computation costs. It scales linearly with the stream size, number of sources, and hidden variables. In contrast, both ICA and PCA have limitations on computation power when the number of genes and observations grows larger. They are also constrained to re-compute the entire microarray matrix when a new experiment is added.
For future work, the proposed method can be investigated in multi-way data analysis. Analyzing multi-way data can reveal more behavior by discovering the correlation between different dimensions. We can capture a multi-linear structure using higher order statistics incrementally. If the data consists of more than two modes, the underlying structures can be detected more efficiently using our incremental approach.