{'fraction_non_alphanumeric': 0.026471750296325564, 'fraction_numerical': 0.016342803778599906, 'mean_word_length': 5.19675050077899, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Using the Revised Cardiac Risk Index to Predict Major Postoperative Events for People With Kidney Failure: An External Validation and Update

Background People with kidney failure have high risk of postoperative morbidity and mortality. Although the revised cardiac risk index (RCRI) is used to estimate the risk of major postoperative events, it has not been validated in this population. We aimed to externally validate the RCRI and determine whether updating the model improved predictions for people with kidney failure. Methods We derived a retrospective, population-based cohort of adults with kidney failure (maintenance dialysis or sustained estimated glomerular filtration rate < 15 mL/min per 1.73 m2) who had surgery in Alberta, Canada between 2005 and 2019. We categorized participants based on RCRI variables and assigned risk estimates of death or major cardiac events, and then estimated predictive performance. We re-estimated the coefficients for each RCRI variable and internally validated the updated model. Net benefit was estimated with decision curve analysis. Results After 38,541 surgeries, 1204 events (3.1%) occurred. The estimated C-statistic for the original RCRI was 0.64 (95% confidence interval: 0.62, 0.65). Examination of calibration revealed significant risk overestimation. In the re-estimated RCRI model, discrimination was marginally different (C-statistic 0.67 [95% confidence interval: 0.66, 0.69]), though calibration was improved. No net benefit was observed when the data were examined with decision curve analysis, whereas the original RCRI was associated with harm. Conclusions The RCRI performed poorly in a Canadian kidney failure cohort and significantly overestimated risk, suggesting that RCRI use in similar kidney failure populations should be limited. A re-estimated kidney failure-specific RCRI may be promising but needs external validation. Novel perioperative models for this population are urgently needed.

Perioperative complications are a major contributor to global morbidity and mortality, and they are the third leading cause of death worldwide. 1 Cardiovascular (CV) etiologies are a major contributor to this postoperative burden, accounting for approximately one-third of these deaths. 2,3 This situation has prompted efforts to identify or predict the risk of major postoperative events for people undergoing surgery, using riskprediction models, which may inform consent for surgery, among other perioperative strategies. 4 Preoperative risk-prediction tools that are used to predict risk of perioperative death and CV events, and are supported by North American guidelines, include the revised cardiac risk index (RCRI), 5 the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) tool, 6,7 and the National Surgical Quality Improvement Program Myocardial Infarction or Cardiac Arrest (NSQIP MICA) tool. 8 The RCRI has been recommended over others for use in Canada for all adults over the age of 45 years, and for those aged 18-45 years with CV disease, who are undergoing elective, noncardiac surgery. 3 The RCRI incorporates 6 criteria based on surgical and comorbidity characteristics of the patient and derives an estimated probability of postoperative myocardial infarction, cardiac arrest, or death. 5 Additionally, the RCRI is used to guide perioperative decision-making. 3 The way that the RCRI addresses kidney disease is problematic for several reasons. First, kidney disease is accounted for in the RCRI, with one point allotted if a patient has a serum creatinine level greater than 177 mmol/L. Although this creatinine level cutoff represents moderate chronic kidney disease (CKD) in many, use of the estimated glomerular filtration rate (eGFR) is strongly recommended to estimate kidney function rather than serum creatinine level. 9 Second, although work is underway to update the RCRI to include eGFR rather than serum creatinine level, 10 people with kidney failure in receipt of maintenance dialysis will not be specifically considered. People with kidney failure (with sustained eGFR less than 15 mL/min per 1.73 m 2 or receiving maintenance kidney replacement therapy) 11 are considered to have similar risk as those with lesser degrees of kidney disease in the RCRI, despite significantly different risk profiles. Accurate perioperative risk estimation is especially important in patients with kidney failure, as they have a greater baseline risk of CV and other comorbid diseases. 12,13 Compared to those without kidney failure, they have an upwards of 16 times higher incidence of major surgery, 14 4-to 10-fold higher odds of postoperative death, 15 and more frequent postoperative CV events and infectious complications. 16 Although people with kidney failure have high surgical risk, the validity of the RCRI has not been evaluated for noncardiac surgery in this population, despite current guideline recommendations for its use. Therefore, we used a population-based kidney failure cohort undergoing noncardiac surgery to externally validate the RCRI and examine whether updating the index could improve its performance in this population.

Study design and source of data
The Alberta Kidney Disease Network database includes person-level linkages of administrative health data, laboratory data, prescription information, and kidney disease-specific data from the province of Alberta, Canada. 17 Alberta has approximately 4.4 million residents, and with universal public health insurance, health data capture is near complete. 17,18 From this database, we derived a retrospective cohort of adults with kidney failure who underwent ambulatory or inpatient surgery. We used this cohort to externally validate and examine the performance of the RCRI for this population. We followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) checklist 19 for prediction-model validation (Supplemental Table S1) and were granted ethics approval by the University of Calgary and the University of Alberta.

Participants
Adults (aged 18 years and older) in Alberta who had inpatient or ambulatory (ie, outpatient) surgical procedures between April 1, 2005 and February 28, 2019 were included if they had preoperative kidney failure. These surgery settings were included given the high risk for both, 20,21 and because the decision of whether to admit people receiving dialysis after had surgery in Alberta, Canada between 2005 and 2019. We categorized participants based on RCRI variables and assigned risk estimates of death or major cardiac events, and then estimated predictive performance. We re-estimated the coefficients for each RCRI variable and internally validated the updated model. Net benefit was estimated with decision curve analysis. Results: After 38,541 surgeries, 1204 events (3.1%) occurred. The estimated C-statistic for the original RCRI was 0.64 (95% confidence interval: 0.62, 0.65). Examination of calibration revealed significant risk overestimation. In the re-estimated RCRI model, discrimination was marginally different (C-statistic 0.67 [95% confidence interval: 0.66, 0.69]), though calibration was improved. No net benefit was observed when the data were examined with decision curve analysis, whereas the original RCRI was associated with harm. Conclusions: The RCRI performed poorly in a Canadian kidney failure cohort and significantly overestimated risk, suggesting that RCRI use in similar kidney failure populations should be limited. A re-estimated kidney failure-specific RCRI may be promising but needs external validation. Novel perioperative models for this population are urgently needed. surgery may be arbitrary if it is arranged simply to accommodate perioperative dialysis. We identified surgical procedures using the Canadian Classification of Health Interventions coding, 22 and excluded diagnostic or radiologic procedures. A comprehensive list of the surgical procedure codes that were considered is included in Supplemental Table S2. We defined kidney failure as an eGFR < 15 mL/min per 1.73 m 2 or receipt of outpatient maintenance hemodialysis or peritoneal dialysis for at least 90 days prior to surgery. Nondialysis participants needed to have at least 2 outpatient serum creatinine measures in the year prior to surgery, and eGFR was estimated using the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equation 23 and averaged per a validated algorithm. 24 People who did not have demographic data available or had emigrated from Alberta within 30 days of their surgery were excluded. Previous kidney transplant recipients were not excluded, so long as they met our kidney failure definition at the time of surgery. Multiple surgeries per participant were eligible, and all variables were determined separately in relation to each surgery (ie, disaggregated from other surgeries in the same person).

Outcome
Our outcome was a composite of 30-day mortality, acute myocardial infarction (AMI), nonfatal cardiac arrest, or ventricular arrythmia. Components were defined per validated algorithms (Supplemental Table S3) and are similar to those reported for the RCRI in the Canadian Cardiovascular Society (CCS) Guidelines on Perioperative Cardiac Risk Assessment for Non-cardiac Surgery. 3

Predictors
The RCRI contains 5 patient variables and 1 surgical variable. We defined each of these variables using our Alberta Kidney Disease Network administrative and laboratory data sources. History of ischemic heart disease, congestive heart failure, and cerebrovascular disease were determined using validated algorithms for International Statistical Classification of Diseases and Related Health Problems, Ninth and Tenth Revisions (ICD-9-CM and ICD-10-CA), codes 25 (Supplemental Table S4). We used an unrestricted lookback period as they are defined in the RCRI. We used linked prescription dispensation data to identify cohort participants who were prescribed insulin therapy for diabetes in the year prior to surgery. As all participants had kidney failure, each was identified as having kidney disease. People with high-risk surgeries (intraperitoneal, intrathoracic, or suprainguinal vascular surgery) were identified using their Canadian Classification of Health Interventions procedure codes (Supplemental Table S4).

Sample size
We used the calculations recommended by Riley et al. 26 for external validation of sample size, and estimated that a minimum of 9861 surgeries with 168 events would be necessary, based on a conservative estimated outcome rate of 1.7% 20 and standard assumptions regarding expected performance estimates and variance of these estimates for external validations. 26 We also estimated the sample size necessary for development and internal validation of an updated model using published recommendations for binary outcomes 27 and the "pmsampsize" 28 module within Stata software, version 17.0 (StataCorp, College Station, TX). Using conservative and liberal outcome estimates of 1.7% and 8.0%, based on previous work, 20,29 and an R 2 of 0.072, 29 we estimated that we would need at least 728 surgeries and 58 events.

Missing data
No data were missing for any of the predictor variables. As serum creatinine measurement (or maintenance dialysis) was a requirement for cohort inclusion, no data were missing for this variable.

Statistical analysis methods
We used Stata software version 16.0 and 17.0 (StataCorp) for all analyses. Overall cohort characteristics were described using medians and interquartile ranges (IQRs) or counts and percentages. We assigned each cohort participant an RCRI score ranging from 1 to 6 (no participant received 0 points). After summarizing the overall distribution of RCRI scores across the cohort, we transformed each score into a probability of major cardiac event or death, per the CCS guideline pooled estimatesdof 3.9% for RCRI score 0; 6.0% for RCRI score 1; 10.1% for RCRI score 2; and 15.0% for RCRI score 3 or greater. 3 We refer to this index as RCRI CCS throughout this article.
We assessed the discrimination of the RCRI CCS for people with kidney failure by estimating the C-statistic and associated 95% confidence intervals (CIs). We assessed calibration of these predictions to observed outcomes in our cohort by examining them in a calibration plot, created with "pmcalplot" 30 in Stata (StataCorp). We completed the same assessments in an analysis limited to inpatient surgeries, to align with the CCS guideline algorithm for cardiac risk assessment.
Next, we fit a prediction model that included the same RCRI variables, but with re-estimated coefficients, per recommended methods for updating prediction models. 31 We used logistic regression to model the 30-day risk of our outcome and included all RCRI variables except for the kidney disease variable, as all participants had this disease; we refer to this model as revised cardiac risk index with kidney failure specific model estimates (RCRI KF ). Clustering of surgeries at the participant level was accounted for by adjusting standard errors in this analysis. As this model was newly reestimated, we internally validated it using bootstrap resampling with the "bsvalidation" 32 module, and estimated the Cstatistic, calibration slope, calibration intercept, and scaled Brier score, with a score close to 0% representing optimal predictive accuracy. Perfect calibration is represented by a calibration slope of 1 and an intercept of 0. 31,33 Risk was not categorized as in the RCRI CCS , and so cohort risk was plotted in deciles across the risk spectrum as recommended in the TRIPOD guidelines. 19 We also examined the sensitivity, specificity, and positive and negative predictive values (PPVs and NPVs) for both the RCRI CCS and the RCRI KF at the predictive thresholds of 5%, 10%, and 15%, which have been identified in the literature on perioperative factors as being important to both patients and care providers. 10,34 Finally, we compared the clinical usefulness of the RCRI CCS vs the RCRI KF using decision curve analysis. This method allows for the estimation of the net benefit of stratification or prediction methods across a range of risk threshold probabilities for an intervention 35,36 . Net benefit is calculated as the weighted balance of true positives and false positives and is compared to a strategy in which no "intervention" occurs, which is context specific. Threshold probabilities become important in such instances, as they are patient-and care provider-specific, and are defined as the risk threshold that would warrant intervention. 36 As an RCRI score of 1 or more corresponds with an estimated outcome risk of 6.0% 3 , and is associated with guideline-supported decisions regarding postoperative cardiac monitoring, we used this threshold to compare the RCRI CCS and RCRI KF .

External validation of RCRI CCS
After assigning risk probabilities to each participant based on their RCRI score, we estimated the C-statistic for the RCRI CCS to be 0.64 (95% CI: 0.62, 0.65; Table 2). Calibration was poor, and the RCRI CCS overestimated risk of our primary outcome (Fig. 2). The expected (ie, predicted) to observed outcome ratio was 3.43. When we limited analysis to inpatient surgeries among people with kidney failure (n ¼ 9917), we found a similar estimated C-statistic of 0.65 (95% CI: 0.64, 0.67; Table 2). Poor calibration and overestimation of risk across the range of predictions still occurred (Supplemental Fig. S1). The expected to observed outcome ratio for inpatient surgeries was 1.31.

RCRI KF model re-estimation and performance
The re-estimated variable coefficients are presented in Table 2. The apparent discrimination was marginally improved with a C-statistic of 0.67 (95% CI: 0.66, 0.69; Table 2) and was identical after internal validation with bootstrapping. The apparent and bootstrap validated scaled Brier scores were both 1.3%. The calibration slope was 1.00 (95% CI: 0.94, 1.10), and the intercept was 0.003 (95% CI: -0.06, 0.07). The calibration curve also demonstrated similar observed and predicted risks across most of the range of predicted risk, though it appeared to overestimate risk at the highest range ( > 10% predicted risk; Fig. 3).

Test characteristics and decision curve analysis
The sensitivity, specificity, PPV, and NPV for both the RCRI CCS and RCRI KF are presented in Supplemental  Table S5. Although the NPVs were high for all thresholds and across models, the PPVs were low. The RCRI CCS had higher sensitivity at the expense of specificity. The RCRI KF had very low sensitivity at these thresholds, with notably higher specificity. We compared the net benefit of using the RCRI CCS and the RCRI KF to guide a potential perioperative strategy that would offer cardiac monitoring in all or in none, and at a threshold of 6.0% (Fig. 4). We observed a net benefit for using the RCRI KF up to a threshold probability of approximately 0.1 (10% predicted risk). In contrast, care informed by the RCRI CCS is associated with net harm at the 6.0% threshold and across almost all threshold probabilities. In the context of these analyses, net potential harm (or a negative net benefit) indicates that the RCRI CCS identifies people as having a perioperative event incorrectly (false positives) more often vs correctly (true positives). Limiting this analysis to inpatient surgeries revealed that both RCRI versions had similar net benefit at the 6.0% threshold as offering the intervention to all participants (Supplemental Fig. S2).

Discussion
Using a retrospective population-based cohort of people with kidney failure undergoing noncardiac surgery, we examined the performance of the RCRI CCS in this external validation study. We found that the RCRI CCS score overestimated the risk of major cardiac events or death within 30 days of surgery, and it may potentially misinform guidance for a perioperative intervention when assessed using decision curve analysis. When the same variables used in the RCRI were used to refit a multivariable risk-prediction model with re-estimated regression coefficients, the resulting RCRI KF demonstrated marginally different discrimination, better calibration, and it has potential to have had superior clinical usefulness in our cohort of people having surgery in both ambulatory and inpatient settings. We found that the RCRI CCS model overestimation led to higher sensitivity at the expense of specificity. In contrast, the RCRI KF had a higher risk of not identifying people who experienced our primary outcome (low sensitivity); both models had low PPV. These findings caution against the use of the RCRI CCS in people with kidney failure and suggest that updated models should be developed and validated to replace the existing RCRI for perioperative risk stratification in this population.
The miscalibration of the RCRI CCS that led to overestimation of risk was notable in our study, and several reasons for this are possible. Misclassification of outcome is important to consider and is most relevant for the nonfatal cardiac components of our composite outcome. Although most administrative data case definitions are highly specific for CV events, they are likely less sensitive than those ascertained with prospective outcome adjudication (and could lead to underestimation). 37 The updated outcome estimates from the CCS guidelines for the RCRI were pooled from studies that used prospective troponin measurement after surgery, and the median C-statistic for included studies was 0.69 (IQR 0.62-0.75). 3 Standardized troponin monitoring may have led to the higher incidence of cardiac outcome detection from the  original studies, suggesting that some of the overestimation could be attributable to lack of identification of asymptomatic AMI in our retrospective study. We also note that for 3 of the 5 studies pooled to inform the CCS-updated RCRI estimates, only AMI was included as an outcome, and we presume that the outcome estimates would be even higher if death were included in all studies and would result in even greater overestimation of risk in our external validation. Also important is recognition of the differences in our RCRI variable definitions, compared to those suggested by guidelines. For example, in our study, "history of coronary artery disease" from administrative health data was dependent on having an AMI diagnosed prior to surgery (Supplemental Table S2). In contrast, the RCRI CCS definition included additional features suggesting coronary artery disease, such as a positive exercise stress test, ischemic chest pain or nitrate use, or a pathological Q-wave on an electrocardiogram. 3 If we had had access to these symptom-or electrocardiogram-based criteria within our data sources, a possibility is that many of our participants would have been assigned higher RCRI scores, which could lead to even greater overestimation of risk. The potential for overestimation of risk has important implications, including misinforming shared decision-making and misguiding implementation of perioperative interventions. Additionally, if using this assessment prior to surgical booking, care providers may be less likely to offer surgery as an option if the estimated risk is deemed too high, contributing to the well-described treatment bias toward people with kidney disease (ie, renalism). 38 Our results raise concerns about use of the RCRI CCS to guide perioperative care for people with kidney failure who are having surgery. Our re-estimated model had improved performance, compared to the RCRI CCS , with better calibration, though only marginal differences in discrimination. We observed an important improvement in estimated net benefit with our updated model, in keeping with other research that has illustrated that clinically important differences in risk stratification can occur with models without an associated improvement in the C-statistic. 35,36,39 As the role of most perioperative risk-prediction models in high-income countries is to inform shared decision-making and perioperative planning, rather than allocate scarce surgical resources, optimizing calibration is the most important goal, and measures such as net benefit can provide clinically relevant evaluation of model use. 31,36 Further, our model had superior estimated clinical utility when used for both ambulatory and inpatient surgery, which is important, as nearly three-quarters of the surgeries were performed in ambulatory settings, with sizeable associated risk for people with kidney failure. 20 Additionally, although presentation of risk-prediction tools as integer-based scores may enable their use across broad settings, 31,40 the wide availability of medical calculators (eg, Calculate by QxMD at https://qxmd.com/, etc. 41 ) has made the use of more mathematically complex regression models a feasible strategy in clinical care. 31,40 These considerations, accompanied by the findings of our external validation, suggest that development and validation of updated models for perioperative risk prediction are warranted for people with kidney failure. To our knowledge, only one other study has evaluated the RCRI in a similar population. In a Brazilian cohort of 325 kidney transplant recipients, the RCRI score was not associated with major adverse cardiovascular events and had discrimination similar to that in the analyses in our study, with an estimated C-statistic of 0.65 (95% CI: 0.49, 0.78). 42 Although that cohort included a smaller group of kidney failure patients who are generally lower-risk surgical candidates, the findings align with our own and suggest that caution is warranted with use of the RCRI in kidney transplantation as well.
Our study is strengthened by the use of a rigorous definition of kidney failure, as we purposefully designed our study inclusion criteria to minimize inclusion of those with acute kidney injury and identify people not in receipt of dialysis.  Additionally, our study benefits from the large size of the cohort of people with kidney failure, necessary for adequate power for external validation, which would be lacking even in large surgical cohorts such as that in the Vascular Events in Noncardiac Surgery Patients Cohort Evaluation (VISION) study based on eGFR criteria. 43 However, our study has limitations related to the nature of administrative health data. As noted, our algorithms to identify predictors and outcomes may be insensitive in some participants. However, for RCRI variable identification, more-sensitive approaches would be expected to result in even higher estimation of risk. Further, we were not able to measure postoperative troponins to identify asymptomatic AMIs, as was done in a large proportion of the VISION study, 44 and as is recommended by relevant guidelines. 3 Given that this practice was recommended only in recent years, only a small proportion of participants would be expected to have these monitored. Roshanov et al. recently examined the RCRI for predicting cardiac events (including myocardial injury after noncardiac surgery) within the VISION study cohort, and found that the RCRI was insensitive and did not sufficiently predict risk. 45 Given the uncertainty related to interpretation of postoperative troponin changes in people with advanced kidney disease, 46 and the likelihood that inclusion would lead to unacceptable loss of specificity for identifying postoperative cardiac events in people with kidney failure, this is less likely to be a major limitation in our study.

Conclusion
In conclusion, among a large Canadian cohort of people with kidney failure undergoing surgery, which we believe is representative of modern healthcare in high-income countries, we found that the RCRI performed poorly for this population, especially if used in an ambulatory setting. Guideline recommendations that suggest use of the RCRI should be interpreted with caution for people with kidney failure. A refit RCRI model improved discrimination marginally, and, notably, improved estimates of calibration and net benefit. As people with kidney failure have unique considerations in the perioperative period, development of new risk-prediction models that incorporate kidney failure-specific variables appears warranted for the perioperative risk stratification of this important population. The RCRI KF , as a newly re-estimated and internally validated model, needs to be evaluated in a geographic or temporally different population (ie, externally validated), and if valid, may serve as a reference for comparison of model performance and clinical usefulness of new models, to help guide and inform perioperative decisions for people with kidney failure.

Data Sharing Statement
This study is based in part on data provided by Alberta Health and Alberta Health Services. The interpretation and conclusions contained herein are those of the researchers and do not necessarily represent the views of the Government of Alberta or Alberta Health Services. Neither the Government of Alberta, Alberta Health, nor Alberta Health Services express any opinion in relation to this study.

Funding Sources
T.G.H. is supported by a Kidney Research Scientist Core Education and National Training Program postdoctoral fellowship (cosponsored by the Kidney Foundation of Canada and Canadian Institutes of Health Research) and the Clinician Investigator Program at the University of Calgary. These funding sources had no role in study design, data collection, analysis, reporting, or the decision to submit for publication.

Disclosures
M.T.J. was the principal investigator of an investigatorinitiated research grant from Amgen, Canada, which is not related to this work. The other authors have no competing interests to disclose.