{'fraction_non_alphanumeric': 0.03353024378068059, 'fraction_numerical': 0.017139529078958316, 'mean_word_length': 5.0842824601366745, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Novel Machine Learning Algorithm Combined With Multivariate Analysis for the Prognosis of Renal Collecting Duct Carcinoma

Objectives To investigate the clinical and non-clinical characteristics that may affect the prognosis of patients with renal collecting duct carcinoma (CDC) and to develop an accurate prognostic model for this disease. Methods The characteristics of 215 CDC patients were obtained from the U.S. National Cancer Institute’s surveillance, epidemiology and end results database from 2004 to 2016. Univariate Cox proportional hazard model and Kaplan-Meier analysis were used to compare the impact of different factors on overall survival (OS). 10 variables were included to establish a machine learning (ML) model. Model performance was evaluated by the receiver operating characteristic curves (ROC) and calibration plots for predictive accuracy and decision curve analysis (DCA) were obtained to estimate its clinical benefits. Results The median follow-up and survival time was 16 months during which 164 (76.3%) patients died. 4.2, 32.1, 50.7 and 13.0% of patients were histological grade I, II, III, and IV, respectively. At diagnosis up to 61.9% of patients presented with a pT3 stage or higher tumor, and 36.7% of CDC patients had metastatic disease. 10 most clinical and non-clinical factors including M stage, tumor size, T stage, histological grade, N stage, radiotherapy, chemotherapy, age at diagnosis, surgery and the geographical region where the care delivered was either purchased or referred and these were allocated 95, 82, 78, 72, 49, 38, 36, 35, 28 and 21 points, respectively. The points were calculated by the XGBoost according to their importance. The XGBoost models showed the best predictive performance compared with other algorithms. DCA showed our models could be used to support clinical decisions in 1-3-year OS models. Conclusions Our ML models had the highest predictive accuracy and net benefits, which may potentially help clinicians to make clinical decisions and follow-up strategies for patients with CDC. Larger studies are needed to better understand this aggressive tumor.


INTRODUCTION
Renal cell carcinoma (RCC) is one of the most common malignant tumors of the genitourinary system. Cancer statistics 2021 reported that about 48780 new cases of RCC in men and 27300 new cases in women will be potentially be diagnosed in the United States during 2021, representing the 6th (5%) and the 9th (3%) most common cancer, respectively (1). Collecting duct carcinoma (CDC), which arises from the principal cells of distal segment of the collecting ducts of Bellini in the renal medulla, is a highly aggressive subtype of RCC (2). CDC is rare in clinical practice and it accounts for 1 to 2% of all renal tumors. Because of its highly malignant biological behavior and most patients are usually diagnosed with local progressive or even distant metastatic diseases, the prognosis is very poor, with a median survival ranging from 10 to 13 months (3)(4)(5). Several studies have reported on small samples of CDC cases and investigated the their clinico-pathological characteristics and prognosis (6,7). However, these studies did not have a large sample size and did not include some non-clinical factors, such as race, marital status and economic status, which have been shown to be risk factors in other researches (8)(9)(10). In addition, there is no model to predict the prognosis of CDC, and an accurate prediction model is important for patient follow-up and treatment decisions (11).
Machine Learning (ML) is an emerging intersection approach that involves several disciplines, which is good at associating multiple discrete variables and accurately predicting outcomes. With the rapid growth of evidence-based medicine as well as vast and complicated medical data acquisition, a more advanced tool is required. ML is emerging as a viable alternative for illness diagnosis and prognostic prediction, with a high predictive performance and a wide range of applications (12,13).
With these considerations, we utilized data from the Surveillance, Epidemiology, and End Results (SEER) database of CDC patients for analysis. This is a publicly available database established by the National Cancer Institute that contains deidentified cancer patients, thus providing a platform for cancer research. Our study aimed to use ML methods to analyze the role of clinical and non-clinical factors on prognosis. In addition, we established a ML model to predict the overall survival (OS) rate of CDC patients.

Data Acquisition and Study Population
This study used the SEER database (https://seer.cancer.gov/) from the National Cancer Institute, which is a freely available cancer registry in the United States. We were granted access to the SEER database files, and all authors adhered to the SEER database's rules throughout the study. Individual informed consents were not required because no personally identifiable information was used in this study. This work was examined and approved by the Jinan University's First Affiliated Hospital's Medical Ethics Committee.
Patients' data were obtained from the SEER 18 Regs Custom Data (with additional treatment fields), Nov 2018 Sub (1975-2016 variation) by using the SEER*Stat 8.3.9.1 software. The selection criteria included patients diagnosed with histologically confirmed CDC of the kidney (site code C64.9-Kidney, Histologic type ICD-O-3 8319/3). Patients were diagnosed between 2004 and 2016, and those who did not undergo surgery or who underwent only partial nephrectomy (PN) and radical nephrectomy (RN) were included. The exclusion criteria in this study were as follows: (a) unknown American Joint Committee (AJCC) on Cancer 6th TNM stage; (b) unknown marital status; (c) unknown race; (d) unknown purchased/ referred care delivery area (PRCDA); (e) regional median family income; (f) unknown laterality of the tumor; (g) unknown tumor size; (h) unknown histological grade; (j) unknown radiotherapy and chemotherapy records and (k) unknown survival months. The downloaded patient data included 243 cases, and the cases who were alive but had survival times of less than 36 months at the follow-up cut-of date were excluded. 215 patients remained in the final cohort and were used in model development.

Variable Selection and Endpoints
In order to take full advantage of the ML algorithm, multiple readily available clinical and non-clinical characteristics were chosen as independent variables for analysis. Several clinicopathological variables that are commonly used in cancer research were selected, including AJCC TNM stage, tumor size, laterality of the tumor, histological grade and surgical method used as well as radiotherapy and chemotherapy records. In addition, the nonclinical variables included age at diagnosis, sex, race, marital status, PRCDA and regional median family income. PRCDA is a geographic area within which either the purchased or referred care is made available by the health service to members who reside within the area and can be used as a rough guide to determine where the patients live. Regional median family income was used to broadly assess the economic status of patients. The endpoints of our predictive models were the 1-, 2-and 3-year OS.

Statistical Analysis and Model Establishment
SPSS (version 22, IBM SPSS software Foundation) and Python (version 3.8.1, Python Software Foundation) was used for all statistical analyses. Firstly, we used a Mann-Whitney U test and a chi-square test to compare continuous and categorical variables, respectively, to assess the distribution of baseline characteristics. For continuous variables, the data were presented as means ± standard deviations (SDs) and for categorical variables, as frequencies (%). Secondly, we calculated the importance of each variable using the XGBoost (XGB) algorithm and assigned weight values to them, with higher scores indicating more importance for the prediction target. XGB is an optimized distributed gradient boosting program and it implements ML algorithms under the Gradient Boosting framework. The univariate Cox proportional hazard model was used to compare the impact of different factors on OS, and then the risk factors were compared with the most important variables that were calculated by the XGB algorithm. Then we used Kaplan-Meier survival estimate to compare survival of patients for each variable. Thirdly, all the patients studied were randomly divided into a training and test set at a ratio of 7:3. The chi-square test was used to analyze the differences between the training and test sets. Several variables with the greatest impact on survival were used for model construction. The training set was used to establish the XGB model and the test set was applied to evaluate it, and we first used 600 trees in XGB in order to build the model.

Model Improvement and Predictive Performance
We analyzed the results of feature importance and risk factors and then selected the following characteristics for model establishment: TNM stage, tumor size, histological grade, age at diagnosis, region and three types of treatment methods. These characteristics were significant in the survival analyses. To ensure that the model was stable, a ten-fold cross-validation was adopted to evaluate its predictive capability. Our model was then tested and adjusted repeatedly, and the parameters were confirmed for the best model obtained.
Performance analysis comprised of three components. Firstly, model discrimination was quantified with receiver operating characteristic (ROC) curve analysis, and its predictive accuracy was assessed using the area under the ROC curve (AUC). Secondly, we used calibration plots to evaluate the performance of the model, which indicated the calibration and how far the predictions of model deviated from the actual event. Thirdly, clinical usefulness and net benefits were assessed with DCA, which could estimate the net benefit of a model by the calculating the difference between the true-and false-positive rates and weighted these by the odds of the selected threshold probability of risks involved. Also, additional ML algorithms such as multivariate logistic regression (MLR) and classifier (Ada) as well as Naive Bayes classification (NBC) were introduced for comparison. ROC curves and calibration plots were used to further evaluate the appropriateness and generalizability of our model.
Other details of the baseline information are tabulated in Table 1.

Important Variables and Survival Analysis
The XGB algorithm was used to identify the feature of importance as judged by the size of the gain value obtained for each variable, with the higher values indicating greater importance for the prediction target: M stage (95 points), tumor size (82 points), T stage (78 points), histological grade (72 points), N stage (88 points), radiotherapy (38 points), chemotherapy (36 points), age at diagnosis (35 points), surgery (28 points), the geographical region where the care delivered was either purchased or referred (21 points), income (19 points), laterality (18 points), sex (17 points), marital status (15 points) and race (12 points) ( Figure 1). With the univariate Cox regression analysis, several high value clinico-pathological variables were defined as risk factors, including age at diagnosis (p=0.002), T stage (p<0.001), N stage (p<0.001), M stage (p<0.001), tumor size (p<0.001), histological grade (p<0.001), surgery (p<0.001), radiotherapy (p<0.001) and chemotherapy (p=0.003). These were also some of the highest scoring variables with respect to the featured importance points allocated. In addition, the univariate Cox regression analysis defined a non-clinical variable, PRCDA, as a risk factor and this was the most important non-clinical feature as calculated by the XGB algorithm (p=0.045) ( Table 2).
According to the Kaplan-Meier survival curves and log-rank tests for categorical variables, T stage, N stage, M stage, tumor size, histological grade, age at diagnosis, radiotherapy, chemotherapy and surgery significantly affected patients' survival. However, race, marital status, sex, laterality, region and income were not significant factors which influenced OS ( Figure 2 and Table 2).

Model Performance
The parameters for the best model performance were confirmed after several tests and adjustments. To determine the accuracy of our models, the ROC curves and calibration plots for the training set (n = 150) and the test set (n = 65) were constructed. The XGB model had the best performance in the training set of 1-, 2-and 3-year OS (AUC=0.895, 0.902 and 0.900, respectively), compared with MLR (AUC=0.819, 0.827 and 0.860, respectively), Ada (AUC=0.821, 0.816 and 0.867, respectively) and NBC (AUC=0.821, 0.769 and 0.807, respectively) ( Figures 3A-C).
With the test set of 1-, 2-and 3-year OS, the accuracy of the XGB model was also higher (AUC=0.855, 0.859 and 0.870, respectively) than all the other three models (Figures 3D-F).
The calibration plots for the training and test sets of 1-, 2-and 3-year OS indicated that the predictive probabilities against observed survival rates showed excellent concordance in the XGB, followed by the Ada and NBC models, respectively. All the models' calibration plots of 1-, 2-and 3-year OS were found to be around the 45-degree ideal line. However, the calibration of the traditional MLR model tended to overestimate the risk across the entire range of predicted probabilities when compared with all the other models ( Figure 4).
DCA of the four algorithms were subsequently constructed in our study ( Figure 5). The y-axis of the decision curve represents the net benefit which is a decision-analytic measure for judging whether any particular clinical decision results in more benefit than harm. Each point on the x-axis represents a threshold probability that differentiates between dead and live patients. This analysis shows that all the models achieved net clinical benefit.

DISCUSSION
CDC is an invasive tumor with aggressive malignancy and very poor prognosis. As the number of nephrectomy procedures increases each year, the detection rate of CDC may rise (14). The American Society of Anesthesiology (ASA) evaluated the prognosis of CDC based on ASA scores, tumor size, distant metastasis, histological grade and lymphovascular invasion and classified it into three risk groups. The 5-year survival rates were 96, 62 and 8% for the low-, medium-and high-risk groups, respectively (15). Because of the rarity of this tumor subtype, its aggressive behavior (advanced stage at presentation and poor prognosis) and the lack of consistent underlying molecular abnormalities, evaluating new treatment options and reaching consensus standards for management has been a difficult task. Several studies have reported cases of CDC treated with chemotherapy or targeted drugs; for example, Tetsuya reported a relatively good prognosis for individual patients with advanced metastatic CDC after receiving combination immunotherapy treatment (16). However, these studies had only a few cases and did not always stratify patients and they lacked prediction of prognosis of the disease (17)(18)(19). As a new type of artificial intelligence algorithm, XGB is efficient and easy to use, and it can deliver high performance and accuracy when compared to other algorithms. It is an improvement over the gradient boosting decision tree (GBDT), which has significant advantages in preventing overfitting, parallel processing, cross-validation and handling of missing values (20). This novel ML algorithm is becoming increasingly popular in the medical arena and it is widely used for disease prediction and early diagnosis (21,22). Based on this consideration, we analyzed the clinico-pathological characteristics of CDC patients, and we established a model to predict their prognosis using a novel XGB algorithm as well as combining clinical and non-clinical variables. To our knowledge, this is the first cohort study based on a relatively large number of CDC patients, reporting several clinical and non-clinical characteristics and factors that may affect the prognosis of CDC patients, which has significant implications for clinicians to deepen their understanding of this disease.
In our study, 164 (76.3%) patients died during follow-up, with a median survival time of 16 months, of which 76 patients were diagnosed with distant metastatic disease. Our cohort had a lower metastatic rate and better prognosis than those reported by other research groups (23,24). In the Kaplan-Meier survival analysis, nine of the 15 selected variables were defined as factors that could affect patient survival, including age at diagnosis, T stage, N stage, M stage, tumor size, histological grade, surgery, radiotherapy and chemotherapy. Surgery could significantly improve the prognosis of patients and those who underwent PN had a better prognosis compared to those who underwent RN. This is probably because only patients with small renal mass and early-stage tumors would accept PN. On the contrary, patients who underwent RN usually had a more advanced stage of the disease and most patients who were inoperable had metastatic disease. Patients who accepted radiotherapy and chemotherapy also had poor prognosis for the same reasons. Physicians may use systemic therapy for those patients with advanced metastases or radiotherapy of the metastatic sites. However, CDC is usually insensitive to these types of treatments (23,25,26). In previous studies, marital and economic statuses were reported to be associated with prognosis, but these were not defined as risk factors in the present study (27,28). This may be due to the extremely poor prognosis of this disease, with a large number of patients dying in a short time. As expected, we found a high degree of agreement between the most important variables calculated by the XGB model and those variables defined as risk factors in the Cox regression analysis. Nine clinico-pathological characteristics were defined as risk factors by univariate Cox regression and were the highest scoring factors as judged by their importance, and they were also found to have significant effects on patient survival. In addition, several studies have reported the impact of non-clinical factors on disease prognosis (29,30). In our study, we found the PRCDA was defined as a risk factor in univariate Cox regression. This parameter was also the highest scoring non-clinical variable and the other non-clinical factors were not statistically significant, although a larger cohort of patients may show to differences. Our novel XGB algorithm was able to assign a value to each variable, including non-clinical ones. This model appears to be better at handling multiple discrete variables, which is difficult to achieve with more traditional methods.
In this study, we selected the nine most important clinical variables as well as a non-clinical variable in order to construct three ML models for the 1-, 2-and 3-year OS of CDC patients. Then we used ROC, calibration plots and DCA to evaluate the performance of our models. The predictive accuracy of our XGB models over three different periods was the highest in both training and test sets (1-year OS AUC=0.895 and 0.855, 2-year OS AUC=0.902 and 0.859, 3-year OS AUC=0.900 and 0.870, respectively). Compared to this, the other ML and MLR models were less accurate. These findings suggest that the MLR and other ML models had weaknesses in their accuracy when analyzing the connection between multiple datasets, whereas the XGB model excelled at properly predicting outcomes from numerous unrelated datasets.
We further evaluated the accuracy of the models with calibration plots and similar results were obtained from these analyses when using the three different periods in both the training and test sets. They predicted probabilities against the observed average survival rates indicating that the XGB model had excellent consistency when compared to other models. In addition, the XGB curves obtained here were closer to the ideal lines. The MLR model tended to overestimate the mortality risks across the entire range of predicted probabilities, which means that this might result in a higher false-positive rate and can lead to over-follow up and overtreatment of some patients.
DCA is a novel approach created by the Memorial Sloan Kettering Cancer Center for assessing the efficacy of therapeutic decisions, which takes into account that clinicians often focus on different goals in different situations during clinical practice. The approach uses the net benefits at different thresholds in order to evaluate the clinical utility of a model (31). In our study, whether in the 1-, 2-or 3-year OS model, within the threshold of most ranges, the net benefit of our XGB model was higher than the other models, which indicates that the value of benefits minus the drawbacks is biggest. The results of the DCA illustrated that the benefits and drawbacks of clinical decisions could be well weighed by our model, with overtreatment and frequent followups being largely avoided. They also suggest a more accurate identification of high-risk patients.
Despite several strengths, this study has certain limitations. Firstly, the data from this study are retrospective and prospective clinical data are needed to provide more reliable evidence for the clinical application of the model. Secondly, the model is based on the SEER database which only contains data from the North American population. There may be gaps in population applicability thereby necessitating the study of a broader population base in future research. Thirdly, due to the limited number of cases in the study cohort, it is possible that some variables did not have sufficient power to show statistical significance.
In summary, this study analyzed the clinico-pathological characteristics and prognosis of CDC patients and used a novel artificial intelligence algorithm to construct three prognostic models. Our model performed very well in predicting the prognosis of CDC patients, and could potentially help clinicians to make more accurate and personalized clinical decisions. The practical use of this model will be to help clinicians predict the OS of CDC patients and so deciding potential follow up strategies. For patients, it could make some who are at high-risk, more vigilant, which may improve their long-term prognosis.

CONCLUSIONS
Our study illustrated the clinico-pathological characteristics of CDC patients and nine clinical and a non-clinical variables were defined as risk factors. Our ML models had the highest predictive accuracy and this may potentially help clinicians to make clinical decisions and follow-up strategies for individual patients with CDC. This study will encourage larger studies to be performed so that we can better understand this aggressive tumor.

DATA AVAILABILITY STATEMENT
The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found in the article/Supplementary Material.

ETHICS STATEMENT
The studies involving human participants were reviewed and approved by Medical Ethics Committee of Jinan University's First Affiliated Hospital. Written informed consent from the participants' legal guardian/next of kin was not required to participate in this study in accordance with the national legislation and the institutional requirements.

AUTHOR CONTRIBUTIONS
LW and ZC conceived and designed the study. YZ and LC provided administrative support. LW, GH, and XQ collected and assembled the data. YH, LW, JL, and LC contributed to data processing, interpretation of results, and drafting. All authors read and approved the manuscript.

ACKNOWLEDGMENT
The authors are grateful to Dr. Dev Sooranna of Imperial College London for English language edits of the manuscript.

SUPPLEMENTARY MATERIAL
The