{'fraction_non_alphanumeric': 0.022625477539191148, 'fraction_numerical': 0.011263338163614807, 'mean_word_length': 4.601365061796717, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Unsupervised Feature Selection for Tumor Profiles using Autoencoders and Kernel Methods

Molecular data from tumor profiles is high dimensional. Tumor profiles can be characterized by tens of thousands of gene expression features. Due to the size of the gene expression feature set machine learning methods are exposed to noisy variables and complexity. Tumor types present heterogeneity and can be subdivided in tumor subtypes. In many cases tumor data does not include tumor subtype labeling thus unsupervised learning methods are necessary for tumor subtype discovery. This work aims to learn meaningful and low dimensional representations of tumor samples and find tumor subtype clusters while keeping biological signatures without using tumor labels. The proposed method named Latent Kernel Feature Selection (LKFS) is an unsupervised approach for gene selection in tumor gene expression profiles. By using Autoencoders a low dimensional and denoised latent space is learned as a target representation to guide a Multiple Kernel Learning model that selects a subset of genes. By using the selected genes a clustering method is used to group samples. In order to evaluate the performance of the proposed unsupervised feature selection method the obtained features and clusters are analyzed by clinical significance. The proposed method has been applied on three tumor datasets which are Brain, Renal and Lung, each one composed by two tumor subtypes. When compared with benchmark unsupervised feature selection methods the results obtained by the proposed method reveal lower redundancy in the selected features and a better clustering performance.


I. INTRODUCTION
Cancer informatics has been characterized by the increasing availability of large data repositories like The Cancer Genome Atlas [Tomczak et al., 2015] and the International Cancer Genome Consortium [Zhang et al., 2011]. One of the most used genome technologies to characterize tumor sub-types is Transcriptomics, which measures gene expression [Lu and Han, 2003]. Each tumor profile is described by more than 20.000 gene expression features which defines a high dimensional input space and further complexity in data analysis. Tumor types presents inner heterogeneity that can be sub-divided in tumor sub-types [Chen et al., 2017] [Xiao et al., 2017] [Chen et al., 2016]. Given the high dimensional space from the input data it is necessary to reduce the dimensionality while preserving the biological interpretation of the system. Moreover, tumor labels like tumor subtype or tumor stage are not always available therefore unsupervised approaches that do not need labeled data are necessary in these cases [Ang et al., 2015]. These reasons motivate us to propose an unsupervised feature selection method for tumor clustering. Feature selection methods are necessary in cancer genomics since they provide a low dimensional representation of the input data. This representation is characterized by a selected subset of the input genes providing interpretability of the results and discarding the rest by following an objective function related to improve a learning task [Li et al., 2018]. In addition, the selected genes can be used to guide biomarker discovery strategies [He and Yu, 2010]. The resulting subspace obtained by a feature selection method is described explicitly by a subset of biological features assuming that the initial feature set contains noisy features that can be discarded. A reduced feature subset has benefits in reducing model complexity and in measuring only a reduced set of biological features [Ang et al., 2015]. In an unsupervised problem it is expected that the selected genes may improve the clustering of tumor profiles. To guide the feature selection process this work proposes a low dimensional and denoised representation of the input data known as latent space which is learned by an unsupervised neural network known as Autoencoder. Then by a Multiple Kernel Learning process a subset of gene features is selected with the objective to approach as much as possible to the learned representation from the autoencoder. Finally by doing clustering using just the selected features it is expected to observe significant clinical attributes associated to each cluster. Although the latent features learned by autoencoders are useful for further clustering of tumors and subtype discovery, these features are a nonlinear combination of the original ones. For this reason these latent features are not useful for biological interpretation since they are not explicit.
The main contribution of this work is an unsupervised method able to select genes with clinical relevance from high dimensional gene expression data without the need of having tumor labels. The proposed unsupervised feature selection method is applied in tumor data from Lung cancer, Renal cancer and Brain cancer. The performance of the proposed method is compared with two unsupervised feature selection methods.

II. RELATED WORK
Reducing the dimensionality of gene expression data can be achieved by feature selection [Ang et al., 2015] and feature extraction methods [Hira and Gillies, 2015]. Feature extraction is the construction of a reduced subset of new features obtained from a linear or nonlinear combination of the initial set of features. Neural Networks have gained popularity for feature extraction and dimensionality reduction lead by the Autoencoder model which is based on nonlinear transformations [Hinton and Salakhutdinov, 2006]. The reduced dimensional space and the extracted features are known as latent space and latent features respectively. Latent features retain salient characteristics from the input data in a low dimensional space [Goodfellow et al., 2016]. Autoencoders have been used in biomedical problems to integrate multi-omic data like gene expression, methylation and microRNA to predict Liver cancer prognosis [Chaudhary et al., 2018]. In a similar way autoencoders have been used to learn meaningful representations from gene expression data of Breast cancer patients and then identify tumor subtypes [Guo et al., 2019]. In addition, Autoencoders have been applied to learn a latent space from somatic mutation data of the pan-cancer landscape showing improvements in the clustering performance [Way and Greene, 2017] [Palazzo et al., 2019]. Moreover, Variational Autoencoders (VAE) have been trained on DNA Methylation data from Lung Cancer patients  and on gene expression data of pan-cancer tumor samples to learn meaningful representations for supervised and unsupervised tasks [Way andGreene, 2017] [Grønbech et al., 2018]. It is clear the important role and capacity of Autoencoders for feature extraction and dimensionality reduction on molecular data from tumors. Multiple Kernel Learning (MKL) have been used for gene selection in supervised problems with the objective to improve the classification between tumor types [Rakotomamonjy et al., 2008] [Du et al., 2017]. Feature selection has been applied also on multi-omic data like Gene Expression, Methylation and miRNA using the Minimum redundancy -maximum relevance (mRMR) [Ding and Peng, 2005] method to predict survival rate on Glioblastoma Multiforme patients . Another study proposes a stable feature selection method for high-dimensional RNA-seq data while applying an ensemble L 1 -norm support vector machines to reduce irrelevant features [Moon and Nakai, 2016] and classify tumor stages of renal clear cell carcinoma. In addition, feature selection by Elastic Net [Zou and Hastie, 2005] has been proposed to select genes linked to the Triple Negative Breast Cancer subtype [Lopes et al., 2018]. The papers described above show the potential and necessity of supervised feature selection methods for gene selection on cancer molecular data. Nevertheless, labeled data is not always available and the selection of genes is needed for unsupervised tasks such as clustering since tumor types may present heterogeneity and each cluster can present different clinical properties. This problem is faced by Unsupervised Feature Selection methods for Clustering [Alelyani et al., 2018]. Multi-Cluster Feature Selection (MCFS) [Cai et al., 2010] is an unsupervised model proposed to select the features that preserves the cluster structure of the original data and has been applied on micro-RNA data [Bandyopadhyay et al., 2015]. Also the Sparse k-Means (SKM) method [Witten and Tibshirani, 2010] has been proposed to weight each feature based on the partition of data and by this way a subset of features is selected by penalizing weights with the L1 norm. Moreover, an unsupervised spectral method (SPEC) [Zhao and Liu, 2007] has been proposed to determine relevant genes on acute lymphoblastic leukemia [Zhao et al., 2010a]. Feature selection and feature extraction methods both have shown potential to reduce the dimensionality of tumor data. In this work we propose a method that combines both strategies by learning a low dimensional latent space by extracting features from the input data and then selecting the gene expression features that approach to the resulting learned latent representation.

III. MATERIALS AND METHODS
This work proposes an unsupervised feature selection method based on Kernel Methods guided by the latent structure obtained from an Autoencoder. The method is divided in tree main steps. First build a target kernel matrix composed by the training samples lying on the latent space of an Autoencoder. Then select features by a multiple kernel learning strategy guided by the kernel built in the previous step as the target representation. Finally with the selected features perform clustering and measure the cluster quality by comparing the enrichment of tumor subtypes on each cluster.

A. Datasets
The proposed method is designed to be used on high dimensional gene expression data from tumor profiles. To evaluate the method three tumor datasets are used: Lung, Renal and Brain cancer. Each type is a separated dataset and is composed by two tumor subtypes samples from the International Cancer Genome Consortium (ICGC) data portal. The Lung Cancer dataset is composed by the projects LUSC-US (Squamus cell subtype) and LUAD-US (Adenocarcinoma subtype) with 478 and 428 tumor samples respectively. The Brain cancer dataset is composed by the projects GBM-US (Glioblastoma) and LGG-US (Lower Grade Glioma) with 159 and 439 tumor samples respectively. The Renal cancer data is composed by the projects KIRP-US (Papillary) and KIRC-US (Clear cell) with 222 and 518 tumor samples respectively. The data matrix for each dataset is X n,d0 with n tumor samples and characterized initially by d 0 = 17640 protein coding genes. The objective is to select a subset of genes p << d 0 to reduce the dimensionality. 1) Pre-processing: To estimate statistically the performance of the proposed method ten independent times 80% of the samples have been randomly selected from the input dataset. The subset of randomly selected samples are used to train the autoencoder and to select the gene features. Each feature has been min-max scaled between 0 and 1 [van den Berg et al., 2006]. Then an univariate filter is applied to reduce the initial number of features from d 0 = 17640 to d = 8820 by ranking the features by variance and only keeping the 50% best ranked. Univariate filter is used to discard low variance features and perform an initial reduction of the high dimensional space. Nevertheless, the subset of d variables preserved remains large and the sample to feature ratio is (n/d) = 0.088 thus the proposed feature selection method is applied at this point.

B. Autoencoders
Autoencoders (AEs) are feed forward artificial neural networks (ANNs). AEs have the objective to learn two functions, an encoder and a decoder. The encoder is a non-linear function that maps the input domain X of n samples and d dimensions to a latent space Z of lower dimension l. On the other side, the decoder is a function designed to reconstruct the samples from the latent space z to the input space. The encoder is forced to learn a function that captures the salient features from X and maps to Z [Goodfellow et al., 2016]. The encoder function is defined as z = f (x) and the decoder asx = d (z). The samples at the latent space are expressed by z whilex represents the reconstructed samples by the decoder function lying on X . During training the autoencoder has to minimize following loss function where L penalizes d (f (x)) when it is different from x. The loss function is computed by the Mean Squared Error (MSE) expressed as Then the encoder F and decoder D functions are expressed as [Kampffmeyer et al., 2017] The encoding function is F (·, W F ) and the decoding D (·, W D ). The expression σ (·) is the activation function of the network. The vectors W and b are the network parameters to learn with the objective to minimize the loss function and represent the weights and biases of the encoder and decoder functions respectively. The optimizer used to learn the parameters of the network is the Adaptive Moment Estimation (Adam) [Kingma and Ba, 2014] which computes depending on the gradient mean an adaptive learning rate to speed up the learning process. In order to force the encoder to learn a useful representation on the latent space and to avoid the AE to just copyx from x different regularization strategies are implemented. First, a regularization term using the L 2 norm is imposed on the weights W F and added to the loss function L. The regularization hyper-parameter is β as follows The regularization term avoid both f and d to have large weights and leads to learn a simpler model, in consequence this reduces the overfitting of the trained model. A second regularization strategy that helps to improve even more the generalization capacity of the model is Batch Normalization (BN) [Ioffe and Szegedy, 2015] which consists to perform normalization at each mini-batch iteration during training.
In this work autoencoders are used to learn meaningful representations from gene expression data from cancer patients. We propose also an architecture for both the Encoder and the Decoder functions. Starting from the Encoder the input layer of dimension d is fully connected to a hidden layer HL 1 of 200 neurons. Then HL 1 is fully connected to a second hidden layer HL 2 with 100 neurons. Finally HL 2 is connected to the latent hidden layer HL z . The latent representation Z has a lower dimension l = 50 in comparison to the input dimension d and describes with less noise the original data. Symmetrically and starting from the latent layer HL z the Decoder Function has one hidden layer HL 3 of 100 neurons followed by one HL 4 of 200 neurons and finally the output layer of dimension d. Using the sample set {z i } in Z a Kernel K z is built and used as target kernel by the following Multiple Kernel Learning step. In the next section Kernel Methods are defined and is explained how features are selected to approach the target representation obtained from the autoencoder.

C. Multiple Kernel Learning
A kernel k is a symmetric function k : X × X → R that meets the condition where X ⊆ R d is a d-dimensional space X , and φ is a mapping function from X to a high dimensional feature Hilbert space H with a dot product such that and H is a Reproducing Kernel Hilbert Space (RKHS). Considering a set of n samples such that (x 1 , x 2 , ..., x n ) from X the Kernel or Gram Matrix K is a n × n matrix with entries defined as K ij = x i , x j . Each position of the Kernel Matrix is expressed as and the kernel function is symmetric and positive semi-definite. By using Kernel functions is not necessary to compute explicitly the mapping φ and the dot products between a pair of samples in the RKHS are computed directly with k (x i , x j ). A Kernel can be thought as a similarity function between vectors samples where at the presence of a pair of orthogonal vectors outputs 0 and at the presence of similar or equal vectors outputs a positive value thus K ij ∈ [0, 1]. This work uses the Gaussian Kernel defined as where σ defines the bandwith of the kernel function. The value of σ parameter is obtained by computing the median of the pairwise distances of the tumor data [Gretton et al., 2005]. The Alignment A(K 1 , K 2 ) between two valid kernels K 1 and K 2 built from a set of samples n is defined as and measures how close both kernels are in terms of finding similarities between pair of samples from n [Cristianini et al., 2002] [ . The operation K 1 , K 1 F means the Frobenious inner product between the two kernel matrices. Kernels can be combined to form more complex functions capable of handling the biological data. Multiple Kernel Learning (MKL) builds a kernel k µ from a linear combination of single kernels k i [Gönen and Alpaydın, 2011] and is expressed as The vector µ represents the weight µ i of each kernel k i and reflects the relative importance of each kernel in the final solution K µ . There are many objective functions to optimize while learning the MKL task, nevertheless in this work the MKL model is built with the objective to maximize the alignment of the resulting kernel K µ and a target kernel K t . Particularly, the target kernel proposed in this method is the one K z built from the samples z i lying at the latent space Z of the Autoencoder and the resulting alignment to optimize is A(K µ , K z ).
To compute the vector of weights µ and the resulting K µ a greedy strategy is used [Pothin and Richard, 2006]. This approach combines just two kernels k 1 and k 2 at each iteration while maximizing the aligment A(K µ , K z ) of the resulting kernel. At each iteration the weights µ 1 and µ 2 of the resulting kernel K µ are obtained by computing the derivative of the alignment and find where it becomes = 0 as optimal condition for each partial derivative. If µ = [µ 1 , µ 2 ] at each iteration then µ 1 and µ 2 are obtained from The condition µ 1 , µ 2 ≥ 0 is needed since only positive linear combination of kernels are valid kernels. Starting from a set of kernels D = [K 1 , K 2 , ..., K d ] the greedy MKL algorithm chooses at the first iteration the kernel K i with highest A(K i , K z ). Then it will start adding at each iteration a new kernel to the solution that improves as much as possible the current alignment until A(K i , K z ) stop increasing. The final vector µ is sparse with only µ i > 0 for every K i selected during the MKL process.

D. Proposed Latent Kernel Feature Selection Method
Given a set of n tumor samples characterized by d gene expression features the data is contained in a X nd expression matrix. It is desired to select a subset of p features from d. To achieve this goal first an autoencoder model is trained and a latent space Z of dimension l is obtained where l << d. Then using the set of samples n projected in the latent space a gaussian kernel K z is built and used as the target kernel. Secondly from X nd a set of d feature-wise kernels are built producing one kernel per feature. Finally by using the MKL model described previously a reduced subset of p kernels is iterativelly selected and combined to build a K µ kernel that increases the alignment A(K µ , K z ). Fig. 1. Pipeline of the proposed method. First starting from the raw data (1) an autoencoder is trained (2) and a latent space learned. Then a Kz kernel built (3) using the sample set projected on the latent space. Finally feature-wise kernels are built (4) and combined by MKL (5) to obtain a Kµ kernel by improving the aligment A(Kµ, Kz). The result is a Xnp matrix characterized by a subset of p features associated to the feature-wise kernels selected by MKL.
Only the feature-wise kernels that increases the alignment A(K µ , K z ) are included in the final kernel K µ . This approach leads to a sparse solution where the non-zero values of the µ vector indicates the feature importance on the result. Features are selected by an unsupervised strategy that best align the representation learned from the autoencoder. We name this method Latent Kernel Feature Selection (LKFS). Figure 1 shows a diagram of the proposed method.

E. Evaluation of selected features
To evaluate the quality of the selected features the Redundancy Rate (RED) [Zhao et al., 2010b] [Yamada et al., 2014] metric is used in this work. This metric measures between the selected features the mean value of absolute correlation and is computed as where ρ ij is the correlation index between the ith and the jth selected features. The RED score takes values between 0 and 1. A RED value close to 0 means low linear correlation between the selected features thus a low redundancy which is a desired result. On the other side when the values of RED are close to 1 means that the selected features are highly redundant which is a non desired output. To evaluate visually the quality of the selected features it is possible to use a non-linear dimensionality reduction method that projects the tumor samples characterized by the selected p features in a two-dimensional representation used only for visualization purposes. Then it is possible to evaluate how the tumor samples are distributed in the new feature subset. The t-distributed stochastic neighbor embedding (t-SNE) [Maaten and Hinton, 2008] is used in this work for this purposes. Another way to evaluate the quality of the selected features is to perform a unsupervised learning task like clustering after feature selection and measure the quality of the obtained clusters. K-means clustering method [Jain, 2010] is applied on the tumor samples characterized by the selected p gene features and each tumor sample x i is assigned to a cluster which implies to assign a cluster label c i . The quality of clusters is evaluated by the Adjusted Rand Index [Rand, 1971] and is done by comparing the cluster label c i obtained with the ground truth y i labels related to the tumor subtype provided as clinical data. It is important to remark that the ground truth labels are never used to select the features and are only used to measure how well the k-means groups the tumor samples. It is computed as where A is the number of tumor sample-pairs assigned to the same cluster and belonging simultaneously to the same tumor subtype, B is the number of tumor sample pairs assigned to different clusters and simultaneously belonging to different tumor subtypes, C is the number of tumor sample pairs assigned to the same cluster but belonging to different tumor subtypes and D is the number of tumor sample-pairs assigned to different clusters and belonging to the same tumor subtype. The Rand Index can be thought as a clustering accuracy and takes values from 0 to 1 where a value close to 0 means a random and non informative clustering results regarding the ground truth clinical labels. When the Rand Index is close to 1 it means that almost every cluster is populated with tumor samples of the same subtype which is a desired score.

F. Baseline methods
To evaluate the performance of LKFS first we stablish a baseline from existing methods. In this work LKFS is compared with the Sparse K-Means (SKM) [Witten and Tibshirani, 2010] and the Spectral Feature Selection (SPEC) [Zhao and Liu, 2007]. Both have been designed to perform unsupervised feature selection for clustering. The SKM method computes via an optimization problem feature weights w = [w 1 , ..., w d ] and applies a lasso-type penalty ||w|| 1 < α to select the most important features while doing k-means clustering. The feature weights are a measurement of the variable importance in clustering. The SKM method is based on the K-means type family of algorithms and assigns a larger weight to the features that have a smaller sum of intra-cluster distances and a smaller or zero weight to the features which a high intra-cluster distance. The other benchmark method is the Spectral Feature Selection for unsupervised learning (SPEC) which is based on spectral graph theory. SPEC uses a pairwise similarity matrix S between samples to build a graph G where each node is a sample and each edge is the similarity measurement. The idea with SPEC is to select the features that are consistent with the graph structure. The objective of SPEC is to select features that gives similar values to samples that are near each other in the graph. A graph G can be built from the pairwise similarity obtained from X. Then the SPEC method makes a feature ranking based on the Normalized Cut of the graph G by using the corresponding Laplacian matrix from the graph. All the presented methods have the number of features to select as an hyperparameter p.

IV. RESULTS
Experiments on each dataset have been conducted ten times by randomly selecting 80% of the data. At each random iteration and in the following order data pre-processing, feature selection and clustering are performed. Then the evaluation by RED and Rand Index is averaged among all random iterations. A set of different number of selected p features is used where p = [10,20,30,40,50]. A set of k clusters are obtained where k = [2, 3, 4, 5]. The first evaluation to be done is the RED score for each subset of selected features p of each method on each dataset.  Figure 2 shows the results of the RED score. It is observed that LKFS has the lowest RED score in all the experiments followed by the SPEC method while SKM has the highest RED. This evidences that the features selected by the LKFS have the lowest redundancy which is a desired result since these are more informative and less redundant. Fig. 3. Two dimensional t-SNE scatter plot for by dataset and method. Each tumor subtype on each dataset is highlighted in blue and orange respectively. Figure 3 shows a two dimension t-SNE scatter plot of each dataset by each method whith p = 50. For the Brain, Lung and Renal datasets LKFS shows how clusters tend to group tumors of the same subtype together in two main clusters. The SKM tend to polarize different tumor subtypes within the same cluster structure and finally SPEC fails to separate effectively the two tumor subtypes of each dataset. Then K-means is applied on the selected features and the quality of clusters measured by the Rand-Index. Figure 4 shows the Rand-Index for different number of k clusters on different number of p selected features for each dataset and for each method. In the Brain dataset the LKFS method clearly outperforms the SPEC and SKM methods for every value of p and k. In both the Renal and Lung datasets the LKFS outperforms the benchmark methods from k = 3 to k = 5. For these two datasets the three methods have a considerably low rand-index with k = 2 since in every case a small and isolated cluster composed by the two subtypes affects the performance and can be observed in Figure 3.

A. Discussion
This work proposes the LKFS method which selects by an unsupervised approach a reduced gene subset from more than 8.000 genes to less than 50 in three different types of cancer datasets. The selection of features is done by improving the alignment between a kernel K z obtained from the latent space Z learned via an autoencoder and by the resulting one K µ after multiple kernel learning on feature-wise kernels. The approach is based on building a target representation of the input data with the latent space of an autoencoder. This target representation has a reduced level of noise and is low dimensional. LKFS selects only the features that align the most to the target representation by kernel alignment. The target representation can be interpreted as a prior distribution that guides the selection process.
To measure the quality of the unsupervised feature selection process the Redundancy score RED is computed on the selected features. In addition a K-means clustering method is applied and the cluster performance is evaluated by the homogeneity of ground truth labels across clusters by the Rand Index. From the experimental results it is observed that LKFS outperforms the SKM and SPEC methods by Rand Index and RED score for almost all the datasets and number of selected features. This suggest that LKFS is able to selects low redundancy features from high dimensional input space that contributes to find well defined clusters composed mainly from one tumor subtype. In contrast, SKM and SPEC select features with higher redundancy which do not contribute enough to build separated clusters neither to group samples from the same tumor subtype together. One of the advantages of LKFS comes from the target representation of the Autoencoder. This representation captures the salient features of the input dataset and then by MKL the selected features will capture approximately the same data structure. Finally, LKFS provides three outputs. The first is the subset of selected features which is considerably reduced in comparison with the original feature set. The second one is the latent space provided by the autoencoder. The latent space serves not only as a target representation for the MKL process but also as a tool for data exploration since it can summarizes in a lower dimensional space the salient features of the original data. The third output is a set of tumor clusters for further subtype discovery. One limitation of LKFS relies in the need to train two models, the AE and the MKL. In this architecture the AE conditions the quality of the selected features.

V. CONCLUSIONS
This work proposes an unsupervised feature selection method named LKFS which can select a considerably reduced subset of meaningful and low redundant features from high dimensional gene expression data. Experimental results show that the proposed method outperforms two unsupervised feature selection algorithms by analyzing the quality of the clusters built on the selected features. LKFS has been evaluated on tumor gene expression datasets from Lung, Renal and Brain Cancer patients and select features that help to identify tumor subtypes without any supervised approach. For this reason LKFS is a useful model for pattern recognition and data mining in a variety of cancer types and high dimensional biological applications. Further work will include multi-omics data fusion approaches in order to consider genomic, proteomic or metabolomic features simultaneously.