{'fraction_non_alphanumeric': 0.03635712824708529, 'fraction_numerical': 0.030629985682143587, 'mean_word_length': 5.1134729603000935, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
External Validation of the 2003 Leibovich Prognostic Score in Patients Randomly Assigned to SORCE, an International Phase III Trial of Adjuvant Sorafenib in Renal Cell Cancer

PURPOSE The 2003 Leibovich score guides prognostication and selection to adjuvant clinical trials for patients with locally advanced renal cell carcinoma (RCC) after nephrectomy. We provide a robust external validation of the 2003 Leibovich score using contemporary data from SORCE, an international, randomized trial of sorafenib after excision of primary RCC. METHODS Data used to derive the 2003 Leibovich score were compared with contemporary data from SORCE. Discrimination and calibration of the metastasis-free survival outcome were assessed in data from patients with clear-cell RCC, using Cox proportional hazards regression, Kaplan-Meier curves, and calculation of Harrell's c indexes. Secondary analyses involved three important SORCE groups: patients with any non–clear-cell subtype, papillary, and chromophobe carcinomas. RESULTS Four hundred seven recurrences occurred in 982 patients in the Leibovich cohort and 520 recurrences were recorded in 1,445 patients in the primary SORCE cohort. Clear discrimination between intermediate-risk and high-risk SORCE cohorts was shown; hazard ratio 2.74 (95% CI, 2.29 to 3.28), c-index 0.63 (95% CI, 0.61 to 0.65). A hazard ratio of 0.61 (95% CI, 0.53 to 0.70) confirmed poor calibration of the two cohorts. Discrimination was observed in secondary populations, with c-indexes of 0.64 (95% CI, 0.59 to 0.69) for non–clear-cell RCC, 0.63 (95% CI, 0.56 to 0.69) for papillary RCC, and 0.65 (95% CI, 0.55 to 0.76) for chromophobe RCC. CONCLUSION The 2003 Leibovich score discriminates between intermediate-risk and high-risk clear-cell and non–clear-cell RCC groups in contemporary data, supporting its use for risk stratification in adjuvant clinical trials. Over time, metastasis-free survival for patients with locally advanced RCC has improved. Contemporary data from adjuvant RCC trials should be used to improve prognostication for patients with RCC.


INTRODUCTION
The Leibovich score, 1 published in 2003, is widely used to guide postnephrectomy prognostication for patients with locally advanced renal cell carcinoma (RCC) 2 and for risk-stratifying patients into adjuvant clinical trials. 2 Leibovich et al developed the score using retrospective data from patients with clear-cell RCC who underwent radical nephrectomy at the US Mayo Clinic between 1970 and 2000. Five features that were significantly associated with time-to-distant metastases (P , .001) comprised the final multivariable model: tumor category (6th TNM 2002), regional lymph-node status, maximum tumor diameter, nuclear grade, and presence of tumor necrosis. For clinical application, risk groups were defined as low (scores 0-2), intermediate (3)(4)(5), and high (6 or higher). Five-year metastasis-free probabilities were reported as 97.1%, 73.8%, and 31.2% respectively. 1 The SORCE trial (ClinicalTrials.gov identifier: NCT00492258), evaluated the effect of sorafenib after nephrectomy and is one of the largest internationally recruiting randomized controlled trial in patients with locally advanced RCC, to date. 3 In SORCE, and now RAMPART (ClinicalTrials.gov identifier: NCT03288532), 4 the 2003 Leibovich score determines participant eligibility and guides their random allocation to trial arms.
Selection of the 2003 Leibovich score for this purpose is supported by its superior discriminative accuracy on direct comparison with several other prognostic scores. 5 Furthermore, the 2003 Leibovich score is simple to calculate. All score components are tumorderived and routinely reported on RCC pathology, negating the need for additional expertise or training. Clinical markers such as patient's performance status are not included in the score, reducing the chance of subjective bias.
An external validation of the 2003 Leibovich score, using data from SORCE participants, was prespecified within the SORCE Protocol (online only). We focused on the intermediate-risk and high-risk patients as they are of specific interest for recruitment to adjuvant clinical trials. SORCE provided a large contemporary data set of individual participant data (IPD) with detailed and long follow-up. Unusually for a validation study, we accessed IPD used to derive the 2003 Leibovich score. 1 By creating closely matched data sets, we were able to compute measures of discrimination and calibration, [6][7][8] to directly compare the performance of the 2003 Leibovich score in the historical and contemporary cohorts. Accordingly, we provide a high-quality evaluation of the Leibovich score's ability to discriminate between patients at intermediate risk and high risk of relapse.
Although the 2003 Leibovich score is used in clinical trials that recruit patients with non-clear-cell RCC, its ability to stratify risk in this group has not been evaluated. Newer prognostic scores [9][10][11] have been developed (including some specifically for non-clear-cell subtypes), but none are commonly used in clinical trials, where straightforward application is key. We present the first exploration of the 2003 Leibovich score's discriminative accuracy within important histologic SORCE subpopulations: any non-clear-cell, papillary-only, and chromophobe-only carcinomas.

Participants: Leibovich Score Calculation
SORCE participants were recruited from July 2007 to April 2013 from 147 centers in seven countries: United Kingdom, Australia, France, Belgium, the Netherlands, Spain, and Denmark, and followed up until July 2019. 3 Only patients with intermediate (3)(4)(5) or high ($ 6) Leibovich scores were included in SORCE. 1,3 Participants with any histology except pure oncocytoma were eligible. Values for components of the 2003 Leibovich score (Data Supplement, online only) were prospectively collected for each participant on random assignment to SORCE.
The 6th TNM 2002 system was used by Leibovich et al and in SORCE. The same nuclear grading system that selects the worst WHO/International Society of Urological Pathology 12 features at each grade was used in both data sets (Data Supplement). In SORCE, this system pragmatically standardized grading across international trial sites and was used for all histologic subtypes including chromophobe and other non-clear-cell RCCs.
The SORCE trial was approved by national regulatory and ethical committees in each participating country and was conducted in accordance with the principles of Good Clinical Practice, the Declaration of Helsinki, and all applicable regulatory requirements and laws. All participants signed an informed consent form before entry into the study.

Participants and Outcomes: Leibovich and SORCE Populations
Two matched cohorts were analyzed. A derivation cohort derived from the 2003 Leibovich data set included patients with clear-cell RCC only and excluded the low-risk group (Leibovich scores 0-2). A validation cohort was derived from intermediate-risk and high-risk clear-cell RCC participants in SORCE. All patients in the 2003 Leibovich data set underwent radical nephrectomy. Partial or radical nephrectomy was permitted in SORCE, reflecting contemporary surgical practice. the date of distant metastases. In the study by Leibovich et al, 1 deaths preceding presumed metastasis were treated as censored observations (C. Lohse, personal communication, April 2020). We defined MFS in the same way for this analysis. We censored time to MFS at 10 years in both cohorts to reflect available follow-up data in SORCE.
Secondary exploratory analyses were conducted in the three SORCE subpopulations: patients with any non-clearcell histology, papillary-only, and chromophobe-only carcinomas.

Statistical Methods
Model validation was performed adhering to transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) guidelines 13 (Data Supplement). The time origin used for both cohorts was date of surgery. A survival analysis allowing for late entry was used, 14 capturing the post hoc nonexposure of a SORCE participant to the risk of an MFS event between surgery and random assignment. In SORCE, 56/1711 (3%) dates of surgery were missing; they were estimated by taking a random selection of 56 values from the distribution of observed intervals between surgery and random assignment.
The performance of the 2003 Leibovich score was assessed using discrimination and calibration. 6,7 Discrimination denotes the ability of a model to distinguish between patients who have and have not experienced an event. Calibration relates to a model's predictive accuracy.
Discrimination was assessed graphically by observing the degree of separation between the Kaplan-Meier curves and by the hazard ratio (HR) between intermediate-risk and high-risk Leibovich risk groups in each cohort. We quantified discrimination according to Harrell's c-index, 8 which denotes the proportion of all usable patient pairs in whom the observed and predicted survival times are concordant. The c-index ranges from 0.5 (performance no better than chance) through to 1 (perfect discrimination).
Calibration measures agreement between predicted and observed outcomes. Good calibration is inferred if Kaplan-Meier curves for risk groups in the derivation and validation cohorts are similar. We quantified calibration through the HR of the indicator variable for the two cohorts (0 5 derivation cohort, 1 5 validation cohort) separately for the two risk groups (0 5 intermediate risk, 1 5 high risk). An HR around 1 suggests accurate calibration.
We also analyzed the ungrouped Leibovich scores 3,4, …,11, to compare the HRs between the individual scores and a base category (taken as score 5 3). We fitted Cox models separately for the derivation and the validation data sets, with each individual score as the explanatory variable and graphed the results.
Furthermore, we analyzed the ungrouped scores as a single entity to compare the discrimination (c-index) of the Leibovich score with that of 2002 TNM staging.
The secondary (exploratory) analyses were conducted with the three SORCE subpopulations using same procedures as with the primary analysis.
All measures were reported with 95% CIs. P values were two-sided.

RESULTS
The 2003 Table 1 describes the demographic, clinical, and histologic characteristics of patients in the 2003 Leibovich data, the derivation cohort, the SORCE data set, and the validation cohort. The validation cohort included more high-risk than intermediate-risk patients (46% v 38%). The validation cohort included 652 (45%) patients who had a laparoscopic nephrectomy and 43 (3%) patients who had a partial nephrectomy, whereas all patients in the derivation cohort underwent radical open nephrectomy (Data Supplement). The median time to MFS in the derivation cohort was 9.2 years, whereas in SORCE, this was not reached within 10 years of follow-up.

Primary Analysis Population: Discrimination and Calibration
Discrimination. Figure 2 presents the results of the validation exercise graphically, showing Kaplan-Meier curves of MFS in the intermediate-risk and high-risk groups for each cohort. Figure 2 shows that discrimination between intermediate-risk and high-risk groups in the derivation cohort is substantial but not entirely maintained in the validation cohort. The c-index in the derivation cohort is 0.67 (95% CI, 0.65 to 0.69) compared with 0.63 (95% CI, 0.61 to 0.65) in the validation cohort (P 5 .01, chi square test).
Discrimination between high-risk and intermediate-risk groups in the derivation cohort, with intermediate-risk as the baseline category, is further indicated by an HR of 3.88 (95% CI, 3.18 to 4.74), compared with 2.74 (95% CI, 2.29 to 3.28) in the validation cohort. Thus, discrimination is maintained in the validation cohort, albeit significantly reduced (P 5 .003, interaction analysis) compared with the derivation cohort.
To assess whether the discrimination of the 2003 Leibovich score degrades over follow-up time, c-indexes at one and 10 years after nephrectomy were compared in both cohorts (Data Supplement). We show that although c-index values for the Leibovich score reduce over time, the difference is small in both data sets.
Calibration. The validation and derivation survival curves for the intermediate-risk and high-risk groups are not aligned (Fig 2), suggesting poor calibration.
Overall, the MFS rate was 26% lower in the validation than in the derivation cohort (HR 5 0.74; 95% CI, 0.65 to 0.85). For the intermediate-risk group, the reduction in MFS rate was 24% (HR 5 0.76; 95% CI, 0.61 to 0.94), compared with 46% (HR 5 0.54; 95% CI, 0.45 to 0.64) in the high-risk group. The results confirm a distinct lack of calibration between data sets. Figure 3 shows that the HRs comparing individual scores with the reference category (Leibovich score 3) increase markedly as the score increases in both the derivation and validation cohorts, reflecting consistently higher discrimination with increasing Leibovich score. We combined groups with scores of 9 and above because very few patients had score 10 or 11, giving unreliable estimates. See the Data Supplement for Kaplan-Meier curves (Fig 1A), HR values (Data Supplement), and c-indexes for each score (Data Supplement). Lower values of c-index and HR for each score group in the validation cohort confirm in detail that discrimination is maintained, albeit attenuated, in the contemporary cohort.

Analysis of Ungrouped Leibovich Scores
In both the validation and the derivation cohorts, the collapse of scores 3-5 and 6-11 into two larger prognostic groups (intermediate-risk and high-risk) results in reduced discrimination compared with the original Leibovich score. This is a compromise to achieve a clinically more useful risk stratification tool.  An HR of 3.88 (95% CI, 3.18 to 4.74) between risk groups was observed in the derivation cohort, compared with 3.21 (CI, 2.05 to 5.03) for SORCE non-clear-cell patients, 2.61 (95% CI, 1.44 to 4.70) for papillary, and 3.88 (95% CI, 1.56 to 9.61) for the chromophobe cohort. Despite smaller cohort sizes with correspondingly larger imprecision, these results highlight the Leibovich score's preserved discrimination in these SORCE subpopulations.

Secondary Analyses: Calibration
Attenuated calibration between the SORCE subpopulations and the derivation cohort for each risk group is shown by   observing the misalignment of the corresponding survival curves (Fig 4). This is quantified by HRs for MFS after fitting a Cox regression model to each risk group separately ( Table 2). Five-year relapse probabilities (Table 3) show improved MFS compared with the corresponding derivation cohort in all SORCE subgroups. The difference is most marked between the high-risk groups.

DISCUSSION
Validation of the 2003 Leibovich score using contemporary IPD from a large international trial represents the highest quality of validation, according to the American Joint Committee on Cancer criteria for model selection. 15 We focused on the intermediate-risk and high-risk clear-cell patients, a group commonly recruited to adjuvant clinical trials. This study confirms that the grouped 2003 Leibovich score, although developed two decades ago, largely retains discrimination in the SORCE validation cohort (c-index 0.63; 95% CI, 0.61 to 0.65) when compared with the derivation cohort (c-index 0.67; 95% CI, 0.65 to 0.69). We therefore support its ongoing use for risk stratification in this setting.
Uniquely, we show that the 2003 Leibovich score discriminates comparably between intermediate-risk and high-risk patients in the non-clear-cell SORCE cohort (cindex 0.64; 95% CI, 0.59 to 0.69). Since the non-clear-cell cohort is limited by inherent variability in clinical trajectories, we explored the two largest non-clear-cell subtypes separately: papillary (c-index 0.63; 95% CI, 0.56 to 0.69) and chromophobe groups (c-index 0.65; 95% CI, 0.55 to 0.76). Although the latter analyses are limited by smaller patient numbers, they indicate negligibly attenuated discrimination compared with the derivation cohort.
Some of the immune-oncology-focused adjuvant RCC trials, including IMMOTION010 (ClinicalTrials.gov identifier: NCT03024996) and KEYNOTE-564, 16 use the TNM staging system for patient random assignment to trial arms. Overall, the simplicity, practical utility, and maintained discrimination in a multisubtype population shown by the 2003 Leibovich score support its standardized use for risk stratification in adjuvant RCC trials in preference to recently published, yet to be widely externally validated, subtypespecific scores. 9,10 We were able to perform a robust calibration analysis using IPD from the original Leibovich Table 3). Overall, better outcomes for patients with locally advanced RCC over time corroborate findings in contemporary literature. 17,18 Improved MFS may be linked to factors such as improved radiologic and pathologic practices over time and the introduction of minimally invasive surgical techniques such as laparoscopic nephrectomy. 19 Differences may additionally reflect an evolution in renal tumor biology over time, driven by changing rates of modifiable risk factors such as obesity and smoking.
Our analysis is not without limitations. First, pathology samples were not centrally reviewed. However, strict guidance for their assessment was provided in the SORCE protocol. Second, patients with low Leibovich risk (score 0-2) were not included this validation, because they are usually cured by surgery or ablation and not usually considered for recruitment to adjuvant trials. We acknowledge that excluding the low-risk group is likely to have resulted in Leibovich Score   In time, it may be possible to improve upon outcome prediction in RCC by adapting prognostic scores to include immunologic or genetic biomarkers that show both prognostic and predictive benefit. An example is the transcriptbased recurrence score, 20 which adds prognostic information when included with the 2003 Leibovich score. However, as it does not predict response to adjuvant treatment and is expensive and complex, it has not been routinely used.
Alongside prognostic and predictive biomarker studies, a pragmatic step will be to refine the 2003 Leibovich score by further unpicking the characteristics known to drive worse outcomes in RCC. A digital pathology review of SORCE tumor samples is underway. This will allow a comprehensive analysis of the heterogeneity among RCC tumor specimens. 21 It may also reveal further granularity within current 2003 Leibovich score features, to enhance the prognostication and the prediction of recurrence for patients with RCC. A practical goal will be to retain as much of the usefulness and simplicity of the original Leibovich score as possible.
In conclusion, the 2003 Leibovich score is a validated prognostic score which, in contemporary data, discriminates between patients with clear-cell RCC at intermediate risk and high risk of disease recurrence. In addition, it comparably discriminates relapse risk in patients with non-clear-cell, papillary, and chromophobe RCCs in our data set. Over time, MFS rates among patients have improved; therefore, clinicopathologic prognostic scores need to be regularly reviewed. With the wealth of data available from recent RCC trials, there is an opportunity to build upon the 2003 Leibovich score to better reflect the changing landscape of RCC.