{'fraction_non_alphanumeric': 0.029347229840921557, 'fraction_numerical': 0.015542146644724813, 'mean_word_length': 4.935431361909929, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The effects of sample size on omics study: from the perspective of robustness and diagnostic accuracy

In the design of a specific omics study, sample size is an essential consideration that represents the balance among investigation ability, financial requirements and ethical issues. And it was reported critical to increase robustness and diagnostic accuracy. In this study, two metabolomics datasets were used to conduct analysis to understand the effects of sample size on omics study and strengthen the importance of it. Overlap and AUC were applied to measure the robustness and diagnostic accuracy respectively. The results showed the positive correlation between sample size and robustness, sample size and diagnostic accuracy. Moreover, the effects of sample size vary in datasets and indexes. So, for a specific dataset, sample size should be determined according to its purpose.


Introduction
The completion of the Human Genome Project in 2003 [1] and the advancement of high-throughput techniques have triggered the massive growth and deluge of biomedical information, providing an unprecedented opportunity for personalized medicine programs that will significantly improve patient care [2]. The age of big data has arrived [3,4]. Of the big data in the field of biomedical, omics data is one of the three primary sources, containing a comprehensive catalog of molecular profiles (e.g. genomic, transcriptomic, epigenomic, proteomic, and metabolomic) [5]. Through the application of high-throughput techniques, omics study can measure the expression levels of tens of thousands of variables simultaneously. Therefore, omics is a magic tool to generate lists of markers and construct predictor by identifying differentially expressed genes, proteins or metabolites between different classes.
However, despite considerable successes of omics study in many fields of disease, such as breast cancer, pancreatic cancer, colorectal cancer and so on, there still exist several problems severely hampering the rapid development of omics study, such as the lack of reproducibility of markers [6] and the lack of consistency in generating classifiers [7]. Specifically, the issue of the stability of markers in omics study has been a problem for quite a long time [8][9][10][11]. For example, among the two lists of markers identified by two prominent studies about breast cancer [12,13], only three genes shared on both. Moreover, the discriminatory power of the predictor generated by one of the most acclaimed prognostic gene lists [13], the 70-gene "Amsterdam signature", dropped to a lower value when validating on another dataset [14]. The problems mentioned above have received great attention and were reported the results of many reasons, such as the heterogeneity in patient cohorts, treatment decisions, technological platforms, approaches used for data analysis, sample size and more. Among these reasons, sample size is an essential consideration that represents the balance among investigation ability, financial requirements and ethical issues [15]. An undersized study means study with insufficient sensitivity that an interesting feature may be missed, resulting in a waste of time and money. Alternatively, when the study is oversized, meaning too many subjects are submitted, the overall costs are increased, especially if it is a potentially harmful or invasive study, there will raise ethical questions [16]. Thus, sample size was suggested critical for improving the predictor accuracy and establishing a reliable list of markers.
In this study, two metabolomic datasets MTBLS28 ESI+ and ESI− [17] from the Metabolites [18] were used to conduct several analyses to understand the effects of sample size in omics study from two aspects, diagnostic accuracy and robustness. The former is the prediction accuracy of a classifier based on marker lists, which could predict the class of a test sample, which is evaluated by receiver operating characteristic (ROC) analysis and measurement of the area under the curve (AUC). The latter is the similarity in content between marker lists generated from various sample sizes, which is measured by overlap [19] between two lists of markers. Different sample sizes are modeled by randomly sampling approach and through the indexes mentioned above, we can see the effects of sample size directly. As a part of omics study, by understating the effects of sample size in metabolomics study, we can extrapolate the results in omics study.

Data pre-processing.
Several factors, like unwanted experimental, biological variations and technical errors, can hamper the identification of markers, resulting in missing values and bias of the data [20]. Since several statistical methods based on complete datasets without bias, it is common and useful to conduct imputation and normalization before further study. KNN (k-Nearest Neighbor) algorithm, which was reported as an efficient method for missing value imputation [21], was applied to reduce the sparsity of the data in this study. And MSTUS [22], was applied for data normalization to remove the unwanted sample-tosample variations. And finally, because of the huge amount of hypothesis tests during the analysis, it is also necessary to conduct a multiple testing adjustment. However, in case of large size of tests and low differential features, the above adjustment will lead to a substantially low power for identifying truly differential features. Data filtering by variance is thus introduced to reduce the size of tests and increase the power [23].

Statistical analysis.
Prior to statistical analysis, random sampling approach and k-means clustering were applied to construct independent training datasets, testing datasets and sub-datasets of various sample size. About 25% the samples (117 lung cancer patients and 134 healthy individuals) of the data were sampled by k-means clustering as the independent testing data, and the remaining data were divided into 6 subdatasets with various sample size, which was varied from 50 to 300 by an increment of 50. Theses sub-datasets was constructed by random sampling without replacement, which was repeated 200 times for each sample size.
For each sub-dataset, first, the differentially expressed features were identified by selecting top 100 features with the lowest p-value of Student t-test. Second, based on the Support Vector Machine (SVM), 6 prediction models were constructed from the training samples. The SVM algorithm used in this study is in the R package 'e1071' (http://cran.r-project.org/web/packages/e1071).

Robustness and Diagnostic Accuracy
Robustness is the fraction of shared features that appear on both two lists of markers which is measured by the similarity of two lists of identified markers. For the biomarker discovery from several datasets, the 100 top ranked features are discovered from the 6 sub-datasets. The value of overlap is then calculated by the part of shared features that appear on both two lists (a and b) of markers. The metric overlap can be calculated using the following equation [19]: Diagnostic Accuracy of each model is evaluated by the receiver operating characteristic (ROC) analysis together with the measurement of the area under the curve (AUC) based on the Support Vector Machine (SVM). The testing dataset generated was then used to evaluate the prediction performance based on the markers by R package 'ROCR' [24].

Increasing sample size improves the robustness of the markers.
As was shown by Fig. 1a and Fig. 1b, the degree of overlap varies in sample size and the larger the sample size, the higher the overlap. For example, the median overlap of MTBLS 28 ESI+ and ESI− with a sample size of 100 are 0.21 and 0.20 respectively, while they increase to 0.52 and 0.43 with 300 samples. If you want to increase the stability of the markers identified to improve the robustness, you need to increase the sample size first. The results preliminarily indicate that an increasing sample size can increase the robustness of the markers. And these results are in concordance with others who study the effects of sample size on genomics [10,25,26].
To further understand the effects of sample size on robustness, we combine Fig. 1 and Fig. 2 to get Fig. 1c. As illustrated by Fig. 1c., while a good reliability of the markers correlates positively with the sample size, the effects of sample size on robustness differ in datasets. For example, to reach a value of 30%, MTBLS 28 ESI+ needs about 150 samples while MTBLS 28 ESI− needs about 200 samples. So, if you want to reach a specific high value of overlap, samples needed for a dataset may be different from another. The results of how sample size affect the diagnostic accuracy are displayed by Fig.2, which are identical with that of on robustness. When there are 100 samples of MTBLS 28 ESI+ and ESI−, the AUC values are 0.73 and 0.68 respectively. And when increasing samples to 200, the AUC values increased as well. Therefore, to guarantee a high diagnostic accuracy, we usually need a large sample size. Fig.2c showed the ROC plot of MTBLS 28 ESI+ and ESI− with a sample size of 300. It suggests that the effect of sample size on diagnostic accuracy also differs in datasets. So, in a specific study, you need to calculate and determine what exactly the sample size is needed to acquire a satisfied result and that is our next work.  Fig.1 and Fig.2 show the positive correlation between sample size and robustness, sample size and diagnostic accuracy respectively. But through the comparation between them, we discover that the degree of sample size effects differs in robustness and diagnostic accuracy and the samples needed to increase overlap and AUC by a specific value are different. As was shown by Fig.1 and Fig.2, less samples are needed to increase overlap than AUC. For example, 100 samples are needed for MTBLS28 ESI+ to increase the value of AUC from 0.66 to 0.76, while only approximate 50 samples are needed to increase the value of overlap by 0.1. The results suggested that for MTBLS ESI+ and ESI−, sample size affects more on robustness than on diagnostic accuracy and thus the sample size determination should depend on the purpose of the study. Further researches are needed to study the differences among these indexes.

Conclusion
In this research, we applied two metabolomics datasets to study the effects of the sample size on omics study from the perspective of robustness and diagnostic accuracy. Two indexes, overlap and AUC, were used to measure the robustness and diagnostic accuracy respectively. From the results, we can conclude that the consideration of sample size is essential to whom wants to conduct a study with high robustness and diagnostic accuracy and different sample sizes are needed to reach a specific value of the index for different datasets. Also, the degree of sample size varies in robustness and diagnostic