{'fraction_non_alphanumeric': 0.02924374319912949, 'fraction_numerical': 0.016231410953935436, 'mean_word_length': 5.049643444871092, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Predicting survival in head and neck cancer: External validation and update of the prognostic model OncologIQ in 2189 patients

Abstract Background Disclosing prognostic information is necessary to enable patients to make well‐informed treatment decisions. OncologIQ is a prognostic model that predicts the overall survival (OS) probability in patients with head and neck squamous cell carcinoma (HNSCC). We aimed to externally validate and update the model with new prognostic factors and translate it to a clinically useful tool. Methods A consecutive retrospective sample of 2189 patients eligible for curative treatment of a primary HNSCC were selected. Discriminative performance was determined using the C‐statistic. Results External validation showed systematic underestimation of OS in the new population, and reasonable discrimination (C‐statistic 0.67). Adding smoking, pack years, BMI, weight loss, WHO performance, socioeconomic status, and p16 positivity to the recalibrated multivariable model, improved the internally validated C‐statistic to 0.71. The model showed minor optimism and was translated in an online tool (www.oncologiq.nl). Conclusions The updated model enables personalized patient counseling during treatment decision consultations.

annually. 1 Treatment is often mutilating and interferes with vital functions such as swallowing and breathing. 2 In addition, diagnosis and treatment have an impact on psychosocial functioning, and the incidence of depression is estimated to be as high as 40%. [2][3][4] Therefore, there is a delicate balance of the trade-offs between quantity and quality of life (QoL) when discussing treatment options. Disclosing prognostic information on both survival probabilities and QoL during these discussions is necessary to enable patients to make well-informed treatment decisions.
Previous research showed that health care providers are not able to accurately forecast cancer patients' lifeexpectancy and tend to overestimate survival. [5][6][7] Over the last years, an internally and externally validated prognostic model named "OncologIQ" has been developed. This model estimates the 1-to 10-year overall survival (OS) chances of patients with primary HNC who are eligible for curative treatment. [8][9][10] While traditionally survival rates are solely based on the TNM-classification of the specific tumor, OncologIQ also includes age, sex, and the Adult Comorbidity Evaluation 27 (ACE-27) as prognostic factors for OS. [8][9][10] With this model, a more personalized estimate of the OS is calculated based on the average treatment effect. These personalized estimations can be used to counsel patients during treatment decision consultations, aiming to support shared decision making (SDM).
Although many prognostic models are developed, few are actually used in clinical practice or evaluated in an impact study. [11][12][13] A reason for this could be that patients', physicians', and caregivers' wishes and preferences are not taken into account in the process or that the model is based on outdated data. Although OncologIQ is used by some head and neck oncologic centers in the Netherlands, the patient data included in the original model is outdated. Due to the improvement of OS in HNC in the past years, 14 it is likely that the original model underestimates survival chances. This brings us to the first aim of this study: (1) to explore how OncologIQ performs in a present-day patient population: external validation. The second aim of this study was: (2) to explore whether adding new prognostic factors would improve OncologIQ's performance: model revision. During our qualitative research on prognostication both patients 15 and doctors' addressed the wish to explore new predictors like smoking and alcohol consumption, in order to further personalize the models' estimations. The third aim was to integrate the updated model in a clinically useful tool. The present study builds on our prior prognostic and qualitative research. 15 Finally, our line of research is described in the discussion which illustrates the steps that were necessary to develop a valued tool for use in clinical practice. . Information on every patient with cancer in the Netherlands is recorded in the Netherlands Cancer Registry (NCR) and often directly used for research purposes. In RONCDOC, the consecutive NCR data of patients with HNC were first merged with corresponding clinical data from the patient record files of the Erasmus MC Cancer Institute. These data were manually checked for each patient, and enriched with additional variables from the patient charts according to an extensive data entry manual. When data of the NCR did not correspond with the data in the patient charts, the patient file data was considered leading. All baseline variables were scored at the time of diagnosis. Finally, the data were cleaned by two independent members of the research team. A log was kept of all changes made. The primary endpoint was OS. Follow-up time was last updated on August 16, 2018 by consulting the Municipal Personal Records Database (MPRD). Final day of follow-up time for a patient was defined as the final date that the patient was confirmed to be alive or the date of death.

| Patient inclusion
Patients who were eligible for curative treatment of a primary squamous cell carcinoma of the glottic larynx, supraglottic larynx, hypopharynx, oropharynx, oral cavity, nasopharynx, and lip were included (n = 2253). Patients who were lost to follow-up (n = 5) or refused curative treatment (n = 59) were excluded. In total 2189 patients were included in the analysis.

| Statistics Netherlands
Statistics Netherlands (SN), also known as Centraal Bureau voor de Statistiek, is a governmental institution that gathers microdata on individuals in the Netherlands. Under certain conditions, these microdata are accessible for statistical and scientific research. The data on education and socioeconomic status (SES) are based on nonpublic microdata from SN and were merged with the RONCDOC dataset.

| Definitions of variables
cTNM was staged according to the 7th American Joint Committee on Cancer (AJCC) edition of the TNM classification of malignant tumors. 16 In case of oropharyngeal tumors, immunohistochemical analysis was performed for tumor suppressor protein p16 (cyclin-dependent kinase 2A). p16 positivity was defined as strong and diffuse nuclear and cytoplasmic immunostaining in >70% of the tumor cells. p16 positive tumors were considered HPV positive, conform the 8th AJCC TNM guideline. 17,18 To distinguish between HPV positive and negative tumors, "HPV+" and "HPV−" was added to the oropharynx category. Lymph node category (cN) of these oropharynx tumors was based on the TNM-7. The cumulative quantity of smoking was defined in packyears (PY) in which one pack year was equal to one packet of 20 cigarettes smoked per day for 1 year. If a patient had stopped smoking for ≥3 months, he or she was considered as a former smoker. If a patient had stopped drinking for ≥6 months he or she was considered as a former drinker. Number of alcohol units per week were scored according to a standardized list: one unit, or 10 grams, of alcohol is equivalent to 12.5 milliliters of pure ethanol. 19 Weight loss in kilograms (kg) was defined as weight loss in the 6 months before diagnosis. WHO performance status, also known as the Eastern Cooperative Oncology Group (ECOG) score, was scored according to the classification published by Oken et al. 20 Marital status was defined as being married or having a durable relationship versus being single or widowed. SES was categorized as (1) employed (self-employed, employee, managing director, etc.) or being a student, (2) unemployed (including receivers of an unemployment benefit, social assistance benefit, etc.), and (3) retired. Successful completed education was categorized according to a national and international classification: (1) lower; less than primary, primary and lower secondary, (2)

| Statistical analyses
Statistical analyses were performed using the Statistical Package for the Social Sciences (version 22 and 25), and R statistical software (Vienna, Austria, version 3.4.0 and 3.5.2) using the R packages mice and rms. Different version of these programs were used due to restrictions in the electronic NS environment. Multiple imputation in R was used to for handling missing data (five iterations) that were missing at random. Education could not be imputed since the missing at random assumption was not plausible.

| External validation
For external validation of the original model, performance was assessed using discrimination and calibration analysis. 23,24 Coefficients of the latest original model are not published and were therefore included in Table S1, Supporting Information. Calibration is the agreement between the predicted probabilities and the observed frequencies, in this case the 5-and 10-year survival. The calibration slope is ideally 1 and reflects whether the effects of the predictors are on average correct. 25 The discriminative performance of OncologIQ was assessed by the Harrell's concordance index (C-statistic). 25 The C-statistic is the most commonly used measure to determine the discriminative performance for binary and time-to-event outcome measures. The C-statistic takes values between 0.5 and 1.0, where 0.5 indicates that the model is not better than chance classification and 1 means perfect discrimination. 26 A C-statistic below 0.6 was considered as poor, a C-statistic over 0.6 as moderate, a C-statistic over 0.7 as good and a C-statistic over 0.8 as strong. 27 Usually, model performance is poorer in external validation compared to the performance in the development data. 13 If this is the case, the model should be updated and adjusted to the conditions in the validation cohort to improve performance. 13,28,29

| Univariable analyses
The Cox proportional hazard regression model was used to calculate the univariable hazard ratios (HR) of overall mortality (OM) with 95% confidence intervals (CI). The log-rank test was used to test significance. P < 0.05 were considered significant.  31 The model was integrated in an online tool using shiny R software.

| Patient population
Baseline characteristics and missing data are summarized in Table 1. Out of the 2189 included patients, 1208 (55.2%) died over 10-year follow-up. The median followup time was 5 years and 5 months. Of the 520 oropharynx tumors, in 420 p16 positivity was not tested. In the 100 tested cases, 41 (41%) had p16 positive tumors (see also Table 1). This is comparable with a recent study that included 1204 patients with oropharyngeal squamous cell carcinoma in the Netherlands of which 32% had a p16positive tumor. 32 Figure 1 shows the calibration plots of the external validation. Predicted probabilities for OS of the original model were systematically lower than the observed frequencies in the new cohort. The calibration slope was 0.88 (95% CI, 0.80-0.98). Previous published discrimination of the original model in the development data showed a C-statistic of 0.73. 9 Discrimination of the original model was tested in the new RONCDOC dataset by applying the coefficients as shown in Table S1. This resulted in a lower discriminative performance with a C-statistic of 0.67.

| Univariable analysis
All candidate predictors showed a univariable significant effect on OM, except for sex (Table 1). Having an HPV+ oropharynx tumor was related to a better survival probability compared to having an HPV− oropharynx tumor. Current smokers had a higher probability of mortality compared to former smokers. Former drinkers had a significant worse survival probability (p = 0.000) compared to current drinkers, while the median alcohol consumption per week in the former group was higher (56 units, IQR 28-105) than in de current drinkers (21 units, IQR . Retired and unemployed patients had a lower survival probability than employed patients. Having finished intermediate or primary education was associated with better survival probability compared to patients who finished lower education.

| Model revision (update)
The coefficients of the variables in the original model were recalibrated per variable and are shown in Table 1.
After recalibrating the original model, the discriminative performance was improved from a C-statistic of 0.67-0.69. Using a stepwise backward selection, the final multivariable model included six new variables. The new prognostic variables smoking, PY, BMI, weight loss, WHO performance, and SES were significant predictors for OS in the multivariable analysis. The HRs for the new variables in the model can be found in Table 1. Adding these prognostic factors improved the discriminative performance of the model from a C-statistic of 0.69 to 0.71. No statistically significant effect of year of diagnosis was found, thus is was not necessary to correct for it. Figure 2 shows the difference in survival probability on an individual level between the original and the updated model. The internally validated C-statistic was 0.71. The model showed only minor optimism, with a shrinkage factor of 0.94.

| Model presentation (online tool)
The final extended model was translated into the existing online tool. 33 In this updated version smoking, PY, BMI, weight loss, WHO performance and SES were added. As patients 15 and doctors expressed different presentation wishes during our qualitative research, two separate pages were made: one with a survival graph (doctors) and one with a pie chart (patients). Figure 3 shows a screenshot of the online prediction model. The coefficients for the full model equation can be found in Table 2. The updated coefficients were multiplied with the shrinkage factor. Shrinked coefficients are also shown in Table 2.

| DISCUSSION
This study aimed to optimize the prognostic model OncologIQ. In total 2189 patients with HNC were included to externally validate and update the original model. In summary, the original model showed systematically underestimation of overall survival in the external validation. Consequently, it was recalibrated. In the updated model the new prognostic variables smoking, PY, BMI, weight loss, WHO performance, and SES were added, which improved its discriminative performance from moderate to good (C-statistic 0.71). The final extended model was integrated in a clinically useful tool, which enables up-to-date personalized prognostic counseling.

| Predictors of survival after HNC diagnosis
Univariable analysis showed statistically significant relations between all new candidate predictors and OM (see Table 1). For example, alcohol consumption at diagnosis showed to be harmful, while earlier studies among patients with HNC showed conflicting results. 34 Having no partner was associated with worse survival, while a higher serum Hb and a higher BMI showed to be protective. These findings agree with earlier research 35,36 and should be taken into account during counseling. Former drinkers had a lower survival probability compared to current drinkers. This was explained by the significant higher reported alcohol intake in the former drinker group.
Although univariable analysis showed a significant relation between OM and all new candidate predictors, in multivariable analysis some of these factors had no extra prognostic value over the factors that were added to the final model. An explanation is that specific predictors, for example, alcohol consumption and smoking, are highly correlated to each other. Therefore the effect of drinking alcohol is implicitly included in the effect of smoking on survival and thus does not improve survival prediction. The discriminative performance of our updated model is comparable with other prediction models that are widely used in clinical practice. For example, the Framingham risk model and the APACHE scores with a C-statistic of little over 0.70. 37,38 Likely reasons for their wide use are the relevance of their predicted outcomes (cardiovascular disease and mortality) and their face validity. 39 The same applies to our model. OncologIQ has high face validity given all the included variables. Furthermore, the prediction of long-term survival chances is highly relevant in HNC care. In addition, the updated model provides substantial different estimated survival probabilities on an individual level (see Figure 2), underlining the necessity to update the original model.

| Model presentation and communication
In accordance with a recent published guide on how to present prediction models in clinical practice, 40 we  15 Although it would be tempting to include treatment modalities in our analyses, this would be invalid to estimate in our observational data. Estimating treatment effectiveness in observational data would provide invalid estimates due to confounding by indication. 41 The golden standard for estimating the effect of treatment is using randomized controlled trial data. Since these data are lacking, we could not include treatment in OncologIQ. The models' survival probabilities are based on the average treatment effectiveness in our population. In the former model, the benefit of chemotherapy was added by an adaptation method. 42 Reason for this was that the development set of the original model hardly included cases in which chemotherapy was added to locoregional treatment, while in the course of time many studies proved the benefit of it. 43,44 During the time patients were included in the RONCDOC database, chemotherapy was part of the standard treatment protocol. Therefore, this hazard is no longer included in the visualization to prevent overestimation of the beneficial effect of chemotherapy. The next step would be to use the model in clinical practice. However, there is little research on how to communicate prognostic model estimates during patient counseling. By organizing focus groups, we recently identified HNC patient preferences on receiving prognostic information. 15 Based on the results, a clinical practice guideline has been developed to support physicians in sharing individualized prognostic information. 15

| Strengths and limitations
A major strength of this study was the large number of consecutive patients with HNC (n = 2189) included in our study and the high number of events (n = 1208) that determines high statistical power. In addition, only five patients were lost to follow-up, as we were able to link our dataset to the MPRD. Given the consecutive design of the RONCDOC cohort, selection bias is eliminated. While data from the NCR is often directly used for analysis, we embedded an extra validation check by comparing the NCR data with the data registered in the patient record files, resulting in a high quality dataset. A limitation of our study is the presence of missing values of some baseline characteristics. However, the percentage of missing values was limited except for To tackle this issue we used multiple imputation, which is the preferred method over a complete case analysis. [45][46][47] Also, all tumors were staged according to the TNM-7. To distinguish between HPV positive and negative tumors, "HPV+" and "HPV−" was added to the oropharynx category based on p16 positivity. However, cN and cT classification of these oropharynx tumors was still based on the TNM-7. This implies that the TNM-7 N2a (metastasis in a single ipsilateral lymph node >3 and ≤6 cm) and N2b (metastasis in multiple ipsilateral lymph nodes ≤6 cm) should not be entered as N1 in OncologIQ but as N2. Future research will focus on staging according to the TNM-8. Furthermore, this model has been developed and externally validated using patient data from the Netherlands and the United States. 8

| Clinical implications and future perspectives
Our overall line of research illustrates a perfect example of the consecutive steps, necessary to develop a valued and clinically useful model 13 that is tailored to patients' and physicians' needs. After developing and validating the original model (step 1), the current disclosure of prognostic information was measured (step 2). We used the results of our qualitative research (steps 2 and 3) as the foundation for the current study. The updated model can be used to support SDM during treatment decision consultations. While many prognostic models in cancer care have been developed, few are used in clinical practice or evaluated in an impact study. 11,13 Therefore, it often remains unclear what the impact of model implementation is on both patients and health care providers. 13 A current trial in our institution is evaluating its clinical impact by measuring decisional conflict 49 among patients with HNC (step 5). In this prospective clinical trial with sequential cohorts, patients who received counseling with and without OncologIQ are compared. Also, a pilot study has been done to examine the effect of the use of OncologIQ in our tumor board meetings. In addition, future external validation of the updated model with the new prognostic variables would be appreciated. Information on survival is not a standalone concept and should be combined with information on QoL. Due to the implementation of our Healthcare Monitor 50 we are currently obtaining QoL data. Future research will focus on modeling QoL together with survival.

| CONCLUSIONS
This study enabled the external validation and recalibration of the prognostic survival model OncologIQ for patients with primary HNC, eligible for curative treatment. Recalibration was necessary due to systematic underestimation of OS in the new patient population. Adding the new prognostic variables smoking, PY, BMI, weights loss, WHO performance, and SES improved OncologIQ's discriminative performance. The model was translated in a clinically useful tool, enabling up-to-date personalized patient counseling during treatment decision consultations. Current research in our institution is evaluating its clinical impact.