{'fraction_non_alphanumeric': 0.03011287394427342, 'fraction_numerical': 0.015668166390401767, 'mean_word_length': 5.124969784868262, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Development and validation of novel nomograms for predicting the survival of patients after surgical resection of pancreatic ductal adenocarcinoma

Abstract Background/Aims Pancreatic ductal adenocarcinoma (PDAC) is associated with high mortality, even after surgical resection. The existing predictive models for survival have limitations. This study aimed to develop better nomograms for predicting overall survival (OS) and cancer‐specific survival (CSS) in PDAC patients after surgery. Methods A total of 6323 PDAC patients were retrospectively recruited from the Surveillance, Epidemiology, and End Results (SEER) database and randomly allocated into training, validation, and test cohorts. Multivariate Cox regression analysis was conducted to identify significant independent factors for OS and CSS, which were used for construction of nomograms. The performance was evaluated, validated, and compared with that of the 8th edition AJCC staging system. Results Ten independent factors were significantly correlated with OS and CSS. The 1‐, 3‐, and 5‐year OS rates were 40%, 20%, and 15%, and 1‐, 3‐, and 5‐year CSS rates were 45%, 24%, and 19%, respectively. The nomograms were calibrated well, with c‐indexes of 0.640 for OS and 0.643 for CSS, respectively. Notably, relative to the 8th edition AJCC staging system, the nomograms were able to stratify each AJCC stage into three prognostic subgroups for more robust risk stratification. Furthermore, the nomograms achieved significant clinical validity, exhibiting wide threshold probabilities and high net benefit. Performance assessment also showed high predictive accuracy and reliability. Conclusions The predictive ability and reliability of the established nomograms have been validated, and therefore, these nomograms hold potential as novel approaches to predicting survival and assessing survival risks for PDAC patients after surgery.


| INTRODUCTION
Pancreatic ductal adenocarcinoma (PDAC) is the most common cancer of the pancreas, accounting for up to 95% of pancreatic cancer cases, and it arises from the malignant transformation of the ductal cells. PDAC is also among the leading causes of cancer-related death. 1 The prognosis of PDAC patients is very poor, and the mortality rate remains quite high. It has been estimated that more than 53 000 incident cases develop each year and that the number of deaths due to PDAC can reach approximately 41 000 deaths yearly. As the incidence and the mortality rates of pancreatic cancer have been on the rise recent decades, PDAC is expected to be the second leading cause of cancer-related death in 2030. 2 Although a number of therapies have become available for PDAC patients, complete surgical resection is considered the only potentially curative treatment. However, the overall 5-year survival rate even after surgical resection is reportedly as low as 20%, with only 5%-7% of patients reaching the median survival time of 25-30 months. [3][4][5][6] Clearly, it is essential to improve the clinical outcomes of PDAC patients after surgery by personalizing treatment for each patient.
A prognostic model could facilitate clinical counseling and guide physicians in making treatment and follow-up plans. Currently, the American Joint Committee on Cancer (AJCC) staging system is widely used as a basis for clinical interventions. However, the AJCC staging system has limitations, as only tumor size and the histological presence are taken into account without consideration of some clinically important prognostic factors, including age, gender, ethnicity, marital status, tumor differentiation, lymph node count (LNC), lymph node ratio (LNR), histology, and grade, differentiation. These limitations may diminish the accuracy of the TNM edition for prognostic evaluation. Thus, a better prognostic model that integrates more clinically important factors is urgently needed.
In this study, we aimed to discover comprehensive factors significantly associated with survival among PDAC patients after surgery and to construct better nomograms for predicting survival, including overall survival (OS) and cancer-specific survival (CSS), in PDAC patients after surgical resection. Such nomograms should improve the prediction of individualized postoperative survival and thus useful for improving the care of PDAC patients following surgical resection.

| Study subjects
In this study, we focused on patients who had been pathologically diagnosed with PDAC after surgical resection. All study subjects were identified from the SEER database (SEER*Stat version 8.3.4) during the period between 2004 and 2015. In the enrollment, the following criteria were used: (a) underwent cancer-directed surgical resection of primary PDAC; (b) complete information was provided about the T/N/M stage and LNC/LNR; (c) histology codes with 8140, 8141, 8142, 8143, 8144, 8145, 8146, and 8147, according to the International Classification of Disease 3rd edition (ICD-O3); and (d) data were provided for OS and CSS. The patients who met the following criteria were excluded from this study: (a) PDAC as secondary cancer; (b) absence or incomplete information about survival, follow-up period, cause of death, or other important clinical characteristics; and (c) PDAC suspected but not pathologically diagnosed. Ultimately, a total of 6323 patients were enrolled. According to machine learning, when the size of the given data is ten thousand and below, the ratio of 7:3 is generally used in the training and test sets, and the ratio of 6:2:2 is usually used in the training, test, and verification sets. Hence, patients in this study were randomly assigned into three cohorts: training cohort (n = 3700), validation cohort (n = 1312), and prospective test cohort (n = 1311) at a ratio of 6:2:2. For group assignment, the data were entered in Excel, and random numbers were generated with the Excel random number generator algorithm. The medical records of the enrolled patients for the following demographical and clinical characteristics were reviewed and analyzed: age, gender, race, marital status, age at diagnosis of PDAC, tumor size, tumor TNM stage, tumor histological findings, cancer-specific death, and vital status. The LNR was defined and calculated by dividing the number of metastatic nodes by the LNC. The histological features were classified as well/moderately differentiated or poorly differentiated/anaplastic differentiated PDAC.
The requirements of institutional review board (IRB) approval and written informed consent were waived due to the retrospective nature of the study of cases in the publicly available SEER database. All authors had signed authorization and were allowed to access the dataset in the SEER database.

| Construction of nomograms
Nomograms were constructed based on the training cohort. The chi-square test was used for comparisons of categorical variables. To determine the optimal number of knots, linear assumptions of the continuous variables, including age, LNR and LNC, were relaxed using restricted cubic spline functions, in which goodness of fit was maximized using the log-likelihood, while information loss was minimized with the Akaike information criterion (AIC). 7

| Validation of nomograms and performance assessment
Model performance was assessed using the concordance index (c-index), in which the values of the C-index ranged 0.5-1, with 0.5 considered no discrimination at all and 1.0 representing perfect discrimination. Calibration was made by comparing the means of predicted survival with those of actual survival using observed Kaplan-Meier estimates. To reduce potential bias, 200-sample bootstrap validation was performed for internal testifying. The values of c-indexes were compared using the compare C package. 8 The precision of the 1-, 2-, and 3-year survival rates predicted by the nomograms was evaluated with time-dependent receiver-operating characteristic (ROC) curve analysis using the time ROC 9 package.

| Kaplan-Meier survival curve and decision curve analyses
The nomograms established in this study were compared with the AJCC8 staging system by risk classification and stratification using Kaplan-Meier survival curves. For risk stratification, Nomo stages were built by ranking scores of the accumulated nomograms scores by deciles to develop 10 risk groups, and each AJCC8 substage was divided by nomo stages to derive three prognostic strata: low, median, and high risk. Mosaic plots were created to determine the distributions between the AJCC8 stages and the Nomo stages. The ranges of threshold probabilities were finalized by decision curve analysis (DCA) 10 for the clinical validity of nomograms. 10 The PASW 18.0 program (SPSS Inc) and the R program (v 3.2.3) using rms 7 were used for statistical analysis. A twotailed P value less than .05 indicated statistical significance.

| Baseline demographic and clinical characteristics of the study subjects
A total of 6323 patients with PDAC were enrolled in this study, and the baseline demographic and clinical characteristics of the study subjects are summarized in Table 1. All characteristics were comparable among the patients in the different cohorts. The median follow-up period was 15 months (range, 1-143 months). Among 2716 (73.4%) patients who died during the period of follow-up, 2389 cancer-specific deaths and 327 non-cancer-specific deaths were identified. The 1-, 3-, and 5-year OS rates were 40%, 20%, and 15%, respectively, and the 1-, 3-, and 5-year CSS rates were 45%, 24%, and 19%, respectively.

| Univariate and multivariate analyses of factors associated with OS and CSS
Univariate and multivariate Cox regression analyses were performed to identify factors significantly correlated with OS and CSS, and the results are presented in Tables 2 and 3. On the univariate analysis, older age (P < .001), marital status (P < .05), larger tumor size (P < .001), more advanced TNM 8th T and N stages (both P < .001), poorly differentiated status (P < .001), metastatic disease (P < .001), PLNC (P < .001), LNC (P < .001), and LNR (P < .001) were significantly associated with OS and CSS ( Table 2). The variables that showed significant association with OS and CSS on the univariate analyses were used for subsequent multivariate analysis to determine the factors independently related to OS and CSS (Table 3).

| Construction and calibration of nomograms for OS and CSS
The significant independent factors identified in the training cohort were used in the construction of nomograms for predicting OS and CSS ( Figure 1). The probability of individual survival was calculated by adding up all scores for each selected variable. The details of labels and points in the nomogram are presented in Tables 4 and 5.
The bootstrap-corrected c-indexes in the training cohort provided good validation (CSS, 0.6425; OS, 0.6404). Calibration plots were generated for the probabilities of 1-, 3-, and 5-year OS and CSS and showed optimal agreement between the survival predicted by the nomogram and the corresponding Kaplan-Meier estimates in the training cohorts (Figure 2), indicating that the two cohorts were calibrated well and the established nomograms were reliable for predicting survival in PDAC patients after surgical resection.

| Comparative analysis of the predictive performance between the nomogram and 8th edition TNM staging systems
The areas under the ROC curves (AUCs) for predicting CSS ranged from 66.77% to 70.03% in the three cohorts from 1 to 3 years. As shown in Table 7, the AUCs for the prediction of OS varied from 66.62% to 69.74% during the same years. Thus, the nomograms exhibited considerable discriminatory capacity.

| Comparative analysis of the predictive performance between nomograms and AJCC stages
First, the nomograms showed the greatest log-likelihoods and c-indexes, together with the smallest values of AIC for CSS and OS prediction in all three cohorts in comparison to the AJCC 8th stages (Table 6). These results suggest that the nomograms were better and more robust for survival prediction than the AJCC stages. Second, according to the points in the nomogram, the nomograms stratified patients into 10 Nomo stages, which showed a better ability to discriminate OS and CSS as compared with the AJCC8 stages illustrated by the Kaplan-Meier curves, especially within 5 years after surgery (Figure 4). The 3-year cumulative survival (Table 8) and HRs (Table 9) of the Nomo stages also confirmed the classification ability of the nomograms. Further analysis (Figures 5 and 6) showed that the nomograms were also capable of stratifying each AJCC8th stage into the following three significant groups for prognosis: low-, medium-, and high-risk groups, demonstrating a good ability for risk stratification. Finally, the dramatic survival heterogeneity between the 8th edition AJCC substages and the Nomo stages was demonstrated intuitively by mosaic plots (Figure 7), indicating the underlying frequencies of staging limitations in the traditional AJCC staging system.

| Decision curve analysis
Decision curve analysis was applied to verify the clinical validity of the nomograms and demonstrated wide ranges of threshold probabilities in all cohorts, showing good clinical applicability of the nomograms for predicting survival of PDAC patients. Additional comparisons between the nomograms and the TNM stages indicated that the net benefit was consistently enhanced in the nomograms, suggesting that the nomograms were superior to the conventional AJCC staging system for the prediction of both OS and CSS among PDAC patients after surgical resection (Figure 8).

| DISCUSSION
Accurate prediction of prognosis is critical for better management of PDAC patients following surgery, and to date, this has mainly relied on the AJCC staging system. However, the predictive accuracy of the existing systems for PDAC, including the updated editions, is unsatisfactory. In fact, even among cases with the same TNM stage, post-surgical survival for PDAC patients remains heterogeneous. 4,11,12 This study, on the basis of a large sample size (6323 cases) spanning a long period (12 years), produced the following major novel findings. (a) Ten independent factors were identified to be significantly correlated with OS and CSS in PDAC patients after surgery. better than that of the 8th edition AJCC staging system as more clinical net benefits were obtained. (e) Our results support that the nomograms established in this study had better performance for the prediction of survival and thus hold potential for clinical application in the predication of survival among PDAC patients after surgical resection. Prognostic nomograms have been widely recognized and well accepted as simple models for prediction of prognosis in which complicated statistical models are represented by elegant graphics. [13][14][15] In addition, prognostic nomograms have been shown to be more comprehensive and accurate with easily measured clinical characteristics and with feasibility of application in clinical practice in comparison with other models. In this study, we integrated more clinically important prognostic factors into the generation of the nomograms for prediction of survival in PDAC patients who had undergone surgical resection than had been used in a number of previous studies. 11

T A B L E 6 Comparison of nomogram
with the AJCC staging system be the most important features that can affect survival after pancreatic resection. Of these characteristics, tumor size was the most important factor associated with OS, especially for tumors <3 cm, 22 for which this category was adopted by the 8th TNM staging system. In addition to tumor size, lymph node status is also recognized as a key prognostic factor in relation with disease-free survival and OS, although various cutoffs have been used in different studies. 23,24 According to the latest TNM staging system, the N stages were stratified into N0, N1, and N2. In addition, tumor differentiation 16 and neural invasion 20 have been found to be independent factors significantly associated with poor clinical outcomes following surgical removal of PDAC. In this study, multivariate logical regression analyses were performed and identified older age, larger tumor size, poorer differentiation, more advanced T and N stages, metastasis, harvested lymph node numbers, and lymph node ratio as independent factors significantly associated with lower OS rates, and thus poorer prognosis, after surgical resection of PDAC. These findings are in agreement with those of a number of previous studies. 11,16-21 Considering   that the surgical procedures performed could be affected by many factors, such as tumor size, location of tumor, condition of patients, and the experience of doctors, we did not include operation as a prognostic factor in this study.

F I G U R E 4 Kaplan
It has been noted that the correlations between some factors and prognosis of patients after surgical removal of PDAC are still matters of debate. For example, the effect of lymph node metastasis on the survival of PDAC patients after surgery has been controversial. 25 In a previous study, 26 the LNR as a prognostic reference factor was taken into account, whereas the N stage of version 8 TNM was not considered as an independent risk factor for both OS and CSS. In this study, multivariate analysis revealed that the LNC, LNR, and the TNM8th N stage were strongly correlated with the survival of patients following surgical resection of PDAC, including OS and CSS. The N staging is currently based on the number of lymph node metastases alone. A previous study 27 demonstrated that patients with two or more positive lymph nodes have a very low 5-year OS rate of 5% and that those individuals with one or less positive lymph nodes have a 5-year OS up to 44%. Additionally, in the same study, the TNM 8th N stage showed that more meticulous stratification of the lymph nodes status was associated with more accurate prognosis prediction. The previous findings suggested that the number of the positive lymph nodes has a significant impact on the prognosis, but it was independently associated with OS and CSS in our study. There is a possibility that the number of positive lymph node metastases may be biased by the LNC. Compared with the number of lymph nodes, the LNC and LNR were more closely related to prognosis. Harvesting of 13-16 lymph nodes was highly recommended for accurate staging and prognosis in a previous study, 28 whereas another previous study 29 considered the number to be 20. The LNR incorporates information concerning positive LNs and an estimate of adequate LNs obtained. An elevated LNR may reflect the progression or tendency of metastasis, and for this reason, was found to correlate with poorer OS and CSS in this study. It may also merit attention in our study that marital status and gender were integrated into the construction of the prognostic nomograms for PDAC for the first time. According to the World Health Organization (WHO), the incidence of pancreatic cancer is higher in men than in women, and this gender difference appears to be even greater in developed countries. It is possible that male patients could be more vulnerable to environmental or genetic factors, and that marriage could make a prognostic difference. 20 Recently, several nomograms to predict the prognosis of PDAC patients have been reported, 26,[30][31][32] and in comparison, the nomograms established in this study show a number of strengths. First, we developed nomograms for the prediction of survival based on the analysis of 6323 patients with PDAC, a larger sample size than used in previous studies. Second, internal and external validations were performed in our study, suggesting agreement between the predicted and actual survival. Third, we also selected covariates based upon the AIC and likelihood rather than statistical significance (P value) to balance model complexity and performance. Fourth, to avoid missing information caused by categorization, we adopted restricted cubic splines for those continuous variables in this study. 7 Fifth, this is the first attempt to build nomograms that can stratify each AJCC stage into three prognostic subgroups to achieve better and more robust risk stratification for the PDAC patients. The risk classification and stratification are important, as they may support a better understanding of the degree of survival heterogeneity within the AJCC stages and F I G U R E 6 Kaplan-Meier curve analysis of risk stratification. Risk stratification of overall survival (OS) for each AJCC8 substages in the (A, C, E, G, I, K) training cohort and the (B, D, F, H, J, L) test cohort. All log-rank P values for trends were <.005, except for (I) (P = .2) may help clinicians identify patients at high risk who require intensified follow-up, ensuring the accuracy of patient counseling and facilitating the planning of personalized treatment. Finally, we do realize that favorable predictive accuracy is not necessarily related to usefulness in clinical practice. If the threshold probability of net benefits is unrealistic, the new prediction model may have limited applicability and may even be harmful compared with existing tools. As such, we conducted DCA in this study to confirm the clinical validity of these nomograms.
As the AJCC staging system is currently used system for assessment of prognosis of PDAC patients, we performed comparative analysis between the newly developed nomograms with this existing system. Based on the data, our nomograms showed better performance, with the higher c-indexes and better AIC values, which were statistically different from those values in the AJCC staging system. In addition, DCA in this study indicated that the net benefit was consistently enhanced with the nomograms, suggesting that the nomograms are superior to the conventional AJCC staging system for the prediction and can be clinically useful. In general, the developed nomograms showed better stability and predictive ability.
Our study has several potential limitations that should be considered. First, serum carbohydrate antigen (CA) 19-9 level, microRNA expression, 33 surgical margin status, vascular invasion, and perioperative chemoradiotherapy were not integrated into the analysis, as this information was not available in the SEER dataset. Second, PDAC patients with distant metastases usually have poor prognosis, with the NCCN guidelines not recommending surgery for patients with M1 stage. 34 In this retrospective study, all PDAC patients, including those with distant metastasis, underwent pancreatic surgery, such as pancreaticoduodenectomy, partial pancreatectomy, and total pancreatectomy. There is a possibility that distant metastasis was detected during the operation, and surgical removal of pancreatic tumors was performed to reduce the tumor load. The patients may then have been treated with adjuvant radiotherapy and chemotherapy after the operation. However, the SEER database did not specify the surgical procedure. As such, these might be the potential factors that could affect the prognosis of patients and make the c-indexes of our nomograms mediocre. Third, owing to the competing risks, the accuracy for predicting OS tended to be lower in comparison with that for CSS in our study. Thus, Cox proportional hazard model with no competing risks for easier interpretation, comparison, and comprehension was chosen. 35 Fourth, our nomograms aimed to select patients who might benefit from further care or additional interventions, such as strengthened treatments, adjuvant therapies, patient consulting, and intensified follow-ups.

| CONCLUSIONS
In this study, we have constructed and validated new nomograms for predicting the 1-, 3-, and 5-year OS and CSS rates after surgical resection of PDAC. These nomograms showed good performance for the prediction of OS and CSS, which was superior to that of the existing, conventional AJCC staging system with an improved net comparison of the nomograms with the 8th version of the AJCC stages for predicting (G) 3-y CSS in the training cohort; (H) 3-y OS in the training cohort. Black horizontal lines represented the assumption that events occurred in no patient within a particular timespan. Red lines represented the assumption that events occurred in all patients within the same time span. Blue lines indicated the net benefit of model prediction. The net benefit per patient of each predictive model within a particular time span was a function of the cohort size with threshold probability, and was computed by addition of the benefit (true positive) and subtraction of the harm (false positive). AJCC, American Joint Committee on Cancer; CSS, cancerspecific survival; OS, overall survival benefit. Intrigued by the exciting findings in this retrospective study, we are planning to conduct a prospective clinical study, in which these nomograms will be evaluated for their performance in predicting survival among PDAC patients after surgery in our department. Therefore, these nomograms hold promise as novel prognosis models for improving the prediction of individualized postoperative survival and improving the care of PDAC patients following surgical resection.