{'fraction_non_alphanumeric': 0.023918820971249095, 'fraction_numerical': 0.011838608359507128, 'mean_word_length': 5.314518179506738, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Prediction of Renal Cell Carcinoma Based on Ensemble Learning Methods

Objective: In recent years, ensemble learning methods have gained widespread use for early diagnosis of cancer diseases. In this study, it is aimed to establish a high-performance ensemble learning model for early diagnosis and classification of renal cell carcinomas. Methods: In the study, hemogram and laboratory data of 140 patients with renal cell carcinoma and 140 patients without renal cell carcinoma were included in the study. The data set includes 27 predictors and 1 dependent variable. The data were obtained retrospectively. In the study, classification performances of machine learning methods and ensemble learning methods were compared. In the study, classification performances of boosting, bagging, voting and stacking ensemble learning methods as well as IB1, IBk, Kstar, LWL, REPTree, Random Forest and SMO classifiers were compared. Results: REPTree classifier provided the highest performance among machine learning methods (Accuracy = 0.867). Among the ensemble learning methods, the Stacking ensemble learning method provided the highest performance in Model 6 (Accuracy = 0.906). Stacking ensemble learning methods performed higher than boosting, voting, bagging ensemble methods and machine learning methods. Conclusion: Stacking ensemble learning methods provide successful results in the early diagnosis of renal cell carcinomas. Stacking ensemble learning methods can be used as an alternative to existing methods for diagnosing renal cell carcinoma. In order to further increase the classification performance of the stacking ensemble learning method, it is recommended to choose a meta classifier suitable for the data set and variable types.


Introduction
Renal cell carcinoma is a type of malignancy tumor that originates from the renal cortex and grows starting from the inner surface of the renal tubules (1). Renal cell carcinoma accounts for approximately 2-3% of all cancer types. In recent years, there has been an increase of 2% in the incidence of renal cell carcinoma disease. Renal cell carcinomas constitute approximately 85% of kidney malignancies (1). Clear cell renal carcinomas, papillary cell renal carcinomas and chromophobe cell renal carcinomas are the most common types of renal cell carcinoma (2). Mostly, in advanced stages, renal region pain, weight loss, high fever, hematuria and defatigation are the physical findings of renal cell carcinoma (3)(4)(5)(6). Physical examination, laboratory tests, ultrasonographic and radiological imaging techniques and renal biopsy are generally used for the diagnosis of renal cell carcinoma. Renal cell carcinomas generally progress asymptomatically even in advanced stages. In the physical examination phase, the possibility of early diagnosis of the disease is low. This situation makes it difficult to diagnose the disease early. Early diagnosis is one of the most important factors for successful treatment in cancer diseases. Important advances have been made for early diagnosis of cancer diseases with technological advances in medical imaging techniques. Another important development for early diagnosis of cancer diseases has been achieved with artificial intelligence and machine learning methods. Machine learning methods have produced solutions for many issues in the field of health in recent years. Machine learning methods are applied in early diagnosis of diseases, risk estimation, genetic sequencing, clinical decision support systems, classification of diseases, and identification of patterns for medical images. Machine learning methods can infer and classify real data by learning the patterns in the training data set and the relationship structures between data. Machine learning methods generally perform the classification of data to diagnose diseases (7)(8). Although machine learning methods are generally very good classifiers, they cannot provide the desired classification performance for some data sets. Overfitting and underfitting problems are the main causes of failure in this classification (9)(10)(11). There are different reasons due to the data set not being suitable for the model or the poor quality of the data set (12). Different solution suggestions have been developed to overcome these problems. One of these solutions is ensemble learning methods. Ensemble learning methods provide a common classification result from the estimates of each classifier by classifying the data of more than one machine learning algorithm separately, rather than classifying the data set by a machine learning algorithm. Thus, common prediction results obtained from more than one machine learning method offer more accurate, more reliable and higher performance compared to the prediction results of a machine learning method (13). Ensemble learning methods are based on the principle that more than one classifier can perform classification with higher accuracy than a single classifier predicts.
A great deal of research has been carried out to predict renal cell carcinomas using machine learning methods. Liu et al. conducted a study on the data they obtained from gene expressions to predict renal cell carcinomas. They used the K-NN algorithm and genetic algorithms to classify renal cell carcinomas. Algorithms have provided successful results in the classification of renal cell carcinomas. Machine learning methods can be successful in the classification of patterns (14). Won et al. classified renal cell carcinoma and other urological diseases with C4.5 algorithm in their study. In the classification process, the data set was obtained from texture patterns. Machine learning methods have shown that they can be used for the early diagnosis of renal cell carcinomas by using patterns (15). Lee (19).
Although the ensemble learning method has emerged recently, many researchers have included ensemble learning methods in their studies. Tan and Gilbert can classify cancer diseases in their study by using ensemble learning methods and gene expression values (20). Luo and Cheng benefited from ensemble learning methods in the diagnosis of breast cancers in their studies. The findings of the model provided higher performance than single classifiers (21). Wang used the Stacking ensemble learning method for the diagnosis of prostate cancer. The ensemble model has produced successful results

Early Diagnosis and Ensemble Learning Methods
Mid Blac Sea J Health Sci 2021;7(1):104-114 in the diagnosis of prostate cancer (22). Onan made use of ensemble learning methods for breast cancer classification in his study. They have observed that ensemble models provide better classification performance than single classifiers. In addition to classifying diseases, ensemble learning method can also be applied to predict the success of treatment methods (23). Shayesteh et al. predicted the results of treatment methods applied to rectal cancer patients with the MRI-based ensemble learning model in their study (24).
In this study, it is aimed to create an ensemble learning model that will serve as a basis for artificial intelligence applications for early diagnosis of renal cell carcinoma. It was aimed to predict renal cell carcinoma with high accuracy performance using the ensemble learning model. In addition, it was aimed to determine suitable ensemble models and classifiers to predict the disease with the highest accuracy performance.

Methods
The data of the study consisted of 140 patients diagnosed with renal cell carcinoma and 140 patients diagnosed other than renal cell carcinoma, who applied to the Urology service of Kahramanmaras Sutcu Imam University Health practice and research hospital. Permission was obtained from the Clinical Research Ethics Committee of Kahramanmaras Sutcu Imam University (Ethics committee approval Protocol No: 2018 / 07-21) to conduct the study. The data were obtained retrospectively. Age, gender, hemogram and biochemical laboratory variables of the patients constitute the data set of the study. The variables and their properties in the data set are given in table 1. Power analysis was used to determine the sample size of the study. Considering the values of Mean ± SD: 12.7 ± 7.2 and Mean ± SD: 15.7 ± 7.6 in the reference studies; it was planned to include α: 0.05 first type error level, β: 0.20 type 2 error level, n: 97 for each group with a power of 0.80 test, and at least n: 194 patients in total. The high number of data in data mining, machine learning and multivariate research increases the learning ability of the model. Therefore, a total of n: 280 patient data, n:140 for each group, were included in the study. The data set of the study includes 1 dependent (target) and 27 independent (predictor) variables.

Figure 1. Importance of variables
The Local Outlier Factor (LOF) algorithm has been applied to the data in order to determine the outliers in the data set. LOF algorithm is a powerful method applied in detecting outliers (25)(26). As a result of the evaluation, outliers were removed from the data set. Standardization was applied to the quantitative variables in the data set.
Feature selection was carried out to determine the contribution of predictor variables to the classification. Findings regarding feature selection are given in figure 1. According to the findings, the contribution of the "pathology" variable to the classification performance was found to be quite low. The "pathology" variable has been removed from the model.
In the study, in addition to machine learning methods, ensemble learning methods were also applied to predict renal cell carcinoma. IB1, IBk, Kstar, LWL, REPTree, Random Forest and SMO classifiers were applied to classify the data in the model. In the model, besides the individual performances of the classifiers, the performances of voting, boosting, bagging and stacking ensemble learning methods were also evaluated. In the Boosting ensemble learning method, the "Adaboost" algorithm has been applied. In the Stacking ensemble learning method, the "Logistic" classifier is used as a meta classifier.
Gridsearch and Multisearch algorithms were used for hyperparameter optimization of classifiers (27)(28)(29). Stacking and voting ensemble methods include more than one classifier in the model. For this reason, 7 different models were created to evaluate the stacking and voting ensemble methods. Classifying information for the models is given in Table 2.

Dataset and Model Evaluation
A 10-fold cross validation method was applied to evaluate the performance of classifiers and ensemble learning methods in the model. Weka 3.9 (Waikato Environmental Knowledge Analysis) and R 3.6.0 open source coded softwares were used to evaluate the data with classifiers and ensemble model (30)(31). Hyperparameter optimization has been applied to increase the performance of the classifiers. The performances of classifiers and ensemble methods were evaluated with accuracy, sensitivity, specificity, AUC (Area of under curve) and precision metrics. The complexity matrix for the metrics used to evaluate the performance of classifiers and ensemble models is given in table 3.

Ensemble Learning Methods
Machine learning methods are constantly evolving to provide higher accuracy performance in classification and prediction operations. Ensemble learning methods are also one of the methods developed so that machine learning methods can do classification with higher accuracy. Ensemble learning methods are based on the principle that more than one classifier will provide higher classification performance together rather than the classification performance of a single classifier.
Common prediction of more than one classifier has higher reliability than the estimation of one classifier (32). The training of the data in the model is performed with more than one machine learning method, and predictions with higher accuracy can be made than only one machine learning method. In ensemble learning models, rather than combining classifiers, a common predictive value is obtained by combining the estimates obtained by the classifiers (33). Different merging and learning techniques can be applied in ensemble learning methods. Choosing the appropriate merging and learning method for the data set and variables provides an increase in performance.

Boosting Ensemble Learning Methods
The Bootstrap Aggregating (Bagging) ensemble learning method is based on the bootstrap sampling method. In the Bagging method developed by Breiman, many different subsets are obtained from the data set with the bootstrap sampling method Subsets obtained from the data set are trained with classifiers (32). All classifiers train different subsets at the same time. Estimates of a large number of classifiers are combined by majority vote. Estimation of the majority among the classifiers is accepted as the estimate of the bagging ensemble learning method (33).

Bagging Ensemble Learning Methods
The Boosting ensemble learning method is based on providing high performance by combining many weak classifiers instead of using only one classifier. In the Boosting ensemble method, the training of data is carried out by iterative operations (35). At every stage of iterative operations, taking into account the mistakes made in the previous stage, the same mistakes are prevented. At the end of the iterative stages, a powerful classifier providing high performance is obtained (34). Adaboost method is a powerful ensemble method suggested by Freud and Shapire (36). In the Adaboost method, weighting is applied at each iterative stage. With the weighting process, the accuracy performance is increased by reducing the error at each stage (33).

Voting Ensemble Learning Methods
Voting is essentially an aggregation technique of classifiers rather than an ensemble learning model. In

Mid Blac Sea J Health Sci 2021;7(1):104-114
the voting technique, estimation of majority among different types of classifiers is accepted as the ensemble estimate. The classification of a single data set by different types of classifiers provides different results in estimates. Different estimates increase the accuracy performance of the results. Increasing the variety of classifiers used for training the data set contributes to the reduction of classification errors (33)(34)(35).

Stacking Ensemble Learning Methods
The stacking ensemble learning method was developed by Wolpert (37). Stacking ensemble learning method, similar to voting ensemble method, is based on training the data set by different types of classifiers. The main difference in the stacking ensemble learning method is that it has a meta classifier. Estimates obtained from different types of classifiers constitute the input data for the meta classifier. The estimate obtained by the meta classifier from the input data is accepted as the estimate of the stacking ensemble learning method (37).

Results
In the study, 140 patients with Renal Cell Carcinoma and 140 patients with different kidney diseases other than Renal Cell Carcinoma were classified using machine learning methods and ensemble learning methods. For classification, variables related to the descriptive and laboratory findings of the patients were included in the model. In the study, classification performances of boosting ensemble learning method, bagging ensemble learning method, voting ensemble method and stacking ensemble learning methods as well as classifiers applied without using ensemble method were evaluated. The performances of the methods were compared in terms of accuracy, sensitivity, specificity, precision, and AUC metrics. Renal cell carcinoma classification performances of machine learning methods and ensemble learning methods are shown in table 4.
According to the findings in the table, the Stacking ensemble learning method provided the highest value in terms of accuracy, sensitivity, specificity, precision and AUC metrics (Model 6 Accuracy = 0.906; Sensitivity = 0.906; Specificity = 0.906; Precision = 0.910 and AUC = 0.944 respectively). The Stacking ensemble learning method provided the highest performance in the classification of renal cell carcinomas. It has been observed that Boosting and Bagging ensemble learning methods do not contribute to the performance of some classifiers. The stacking ensemble learning method provided the highest performance in Model 6. Model 6 consists of IB1, IBk, Kstar, LWL, REPTree, Random Forest classifier. The "Logistic" classifier, determined as a meta classifier for the Stacking ensemble learning method, contributed to the classification performance of the ensemble model. The performances of the ensemble methods are shown in figure 2.

Discussion
The incidence of renal cell carcinomas has increased significantly in recent years. Early diagnosis of renal cell carcinomas increases the success rate of the disease in treatment. Medical imaging techniques, laboratory tests, biopsy and physiological examination are important for diagnosing the disease. Recently, artificial intelligence technologies have been applied in addition to traditional methods for early diagnosis of cancer diseases. Machine learning methods are important artificial intelligence techniques applied for early diagnosis and classification of diseases. There have been many studies using machine learning methods for early diagnosis of cancer diseases (38)(39)(40)(41).
Machine learning methods provide successful results for early diagnosis and classification of cancer diseases with high prevalence. Skin cancer, lung cancer, breast cancer, prostate cancer, stomach cancer, cervical cancer and colorectal cancer are some of the cancer diseases with the highest prevalence in the population (42). Although many studies have been conducted on cancer diseases with a high prevalence, machine learning methods have been applied less for cancer types with low prevalence. It is easier to reach a high number of data in cancer diseases with a high prevalence. The number of data is low in cancer diseases with low prevalence. In order for machine learning methods to be applied successfully, the number of samples in the data set should be high. In cases where the number of samples in the data set is low, the machine learning model cannot learn the pattern between data and variables well. This situation causes underfitting problem. The model shows low classification performance when the number of samples in the data set is small.  In ensemble learning methods, more than one classifier functions in the model. The overall classification error of the ensemble model is reduced by combining the estimates of each classifier. Thus, higher performance and lower errors can be obtained in the ensemble learning method compared to only one classifier (13). Ensemble learning methods can provide higher performance than machine learning methods in the classification and early diagnosis of cancer diseases with low prevalence.

Early Diagnosis and Ensemble Learning Methods
In this study, the classification performance of ensemble learning methods and machine learning methods were evaluated for the prediction of renal cell carcinoma. Classifiers IB1, IBk, KStar, LWL, Random Forest, REPTree and SMO were included in the model. In the model, besides the performance of each classifier, their performances in bagging, boosting, voting and stacking ensemble learning methods are also evaluated. According to the findings of the study, REPTree classifier achieved the highest performance among the classifiers which was not subjected to the ensemble method. The accuracy value of REPTree classifier was obtained as = 0.867 and AUC value = 0.932. In the study, the stacking ensemble learning method achieved the highest performance among the ensemble learning methods.  (46). Lin et al. worked on decision tree-based classifiers to predict renal cell carcinomas with tomography images in their studies. They compared the performances of machine learning methods in the model. The highest performance value was achieved as AUC 0.87 (18). The random forest algorithm generally exhibits high performance in the classification of renal cell carcinomas. In this study, REPTree classifier achieved the highest performance among machine learning methods where ensemble learning methods were not used.
In the study, the performances of ensemble learning methods as well as machine learning methods were evaluated to predict renal cell carcinomas. The performance of the boosting, bagging, voting and stacking ensemble methods are evaluated in the model. The REPTree classifier provided the highest performance in the Boosting ensemble learning method. In the Boosting ensemble learning method, the accuracy value of the REPTree classifier was obtained as 0.863. The IBk and REPTree classifiers provided the highest performance in the bagging ensemble learning method. In the bagging ensemble learning method, the accuracy value of both classifiers was obtained as 0.847. Model 7 provided the highest performance in the Voting ensemble learning method. The accuracy value of Model 7 was obtained as 0.878. Model 6 provided the highest performance in the stacking ensemble learning method. The accuracy value of Model 6 was obtained as 0.906. According to these values, Stacking ensemble learning method provided the highest performance among machine learning and ensemble learning methods. Ensemble learning methods generally show higher classification performance than machine learning methods. Classification performances in our study are consistent with the literature. Mohebian (49). Farahani, Ahmadi and Zarandi in their study, applied ensemble learning methods to detect lung nodules on tomography images. Support vector machine, K-NN and artificial neural networks are applied in the model (50).
In our study, ensemble learning methods showed a higher classification performance compared to machine learning methods in cancer types with low prevalence such as renal cell carcinoma. It has been observed that the ensemble model significantly contributes to the reduction of classification error. In order to increase the classification performance in ensemble learning method, it is very important to select the appropriate ensemble model, merging model and appropriate classifiers for the data set. Choosing the right meta classifier in the stacking ensemble method improves the classification performance. Ensemble learning methods can be successfully applied for the classification and early diagnosis of renal cell carcinomas. There are very few studies in the literature regarding the early diagnosis of renal cell carcinomas by ensemble learning methods. Our study will make a significant contribution to the literature in this field.

Conclusions
In this study, the classification performances of ensemble learning methods and machine learning methods were evaluated for the classification of renal cell carcinomas. High accuracy performance ensemble learning model has been established for early diagnosis of renal cell carcinoma. IB1, IBk, Kstar, LWL, REPTree, Random Forest and SMO classifiers were applied in the study. These classifiers are also included in ensemble learning methods. In the study, boosting, bagging, voting and stacking ensemble learning methods were applied. Stacking ensemble learning method provided the highest classification performance. Model 6 (IB1, IBk, Kstar, LWL, REPTree, Random Forest) provided the highest performance in the stacking ensemble learning method. The accuracy value of Model 6 was 0.906, the sensitivity value was 0.906, the specificity value was 0.906, the precision value was 0.910, and the AUC value was 0.944. In our study, renal cell carcinomas were successfully classified with high performance using ensemble learning methods. The model created using the ensemble learning method can be successfully applied for early diagnosis of renal cell carcinomas. In our study, stacking ensemble learning methods provided higher classification performance than machine learning methods. Stacking ensemble learning method can diagnose renal cell carcinomas early with high accuracy performance using hemogram and laboratory findings. Including more variables in the model in ensemble learning methods can increase the classification performance of the model. Increasing the number of samples in the dataset may contribute to the increase in classification performance. It is recommended to choose the appropriate meta classifier for the dataset in the stacking ensemble learning method. In order to increase the classification performance in ensemble learning methods, it is recommended to include different types of classifiers (decision tree-based, function-based and distance-based) into the model.