{'fraction_non_alphanumeric': 0.028275988286969252, 'fraction_numerical': 0.021001098096632503, 'mean_word_length': 5.068017767906719, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Analysis of a Machine Learning–Based Risk Stratification Scheme for Chronic Limb-Threatening Ischemia

IMPORTANCE Valid risk stratification schemes are key to performing comparative effectiveness research; however, for chronic limb-threatening ischemia (CLTI), risk stratification schemes have limited efficacy. Improved, accurate, comprehensive, and reproducible risk stratification models for CLTI are needed. OBJECTIVE To evaluate the use of topic model cluster analysis to generate an accurate risk prediction model for CLTI. DESIGN, SETTING, AND PARTICIPANTS This multicenter, nested cohort study of existing Project of Ex Vivo Vein Graft Engineering via Transfection (PREVENT) III clinical trial data assessed data from patients undergoing infrainguinal vein bypass for the treatment of ischemic rest pain or ischemic tissue loss. Original data were collected from January 1, 2001, to December 31, 2003, and were analyzed in September 2021. All patients had 1 year of follow-up. EXPOSURES Supervised topic model cluster analysis was applied to nested cohort data from the PREVENT III randomized clinical trial. Given a fixed number of clusters, the data were used to examine the probability that a patient belonged to each of the clusters and the distribution of the features within each cluster. MAIN OUTCOMES AND MEASURES The primary outcome was 1-year CLTI-free survival, a composite of survival with remission of ischemic rest pain, wound healing, and freedom from major lower-extremity amputation without recurrent CLTI. RESULTS Of the original 1404 patients, 166 were excluded because of a lack of sufficient feature and/or outcome data, leaving 1238 patients for analysis (mean [SD] age, 68.4 [11.2] years; 800 [64.6%] male; 894 [72.2%] White). The Society for Vascular Surgery Wound, Ischemia, and Foot Infection grade 2 wounds were present in 543 patients (43.8%), with rest pain present in 645 (52.1%). Three distinct clusters were identified within the cohort (130 patients in stage 1, 578 in stage 2, and 530 in stage 3), with 1-year CLTI-free survival rates of 82.3% (107 of 130 patients) for stage 1, 61.1% (353 of 578 patients) for stage 2, and 53.4% (283 of 530 patients) for stage 3. Stratified by stage, 1-year mortality was 10.0% (13 of 130 observed deaths in stage 1) for stage 1, 13.5% (78 of 578 patients) for stage 2, and 20.2% (105 of 521 patients) for stage 3. Similarly, stratifying by stage revealed major limb amputation rates of 4.2% (5 of 119 observed major limb amputations in stage 1) for stage 1, 10.8% (55 of 509 patients) for stage 2, and 18.4% (81 of 440 patients) for stage 3. Among survivors without a major amputation, the rates of CLTI recurrence were 9.2% (11 of 119 observed recurrences in stage 1) for stage 1, 24.9% (130 of 523 patients) for stage 2, and 29.6% (132 of 446 patients) for stage 3. (continued) Key Points Question At the time of chronic limbthreatening ischemia (CLTI) diagnosis, how are increasing patient risk, wound severity, and degree of atherosclerotic occlusions associated with CLTI-free survival? Findings This cohort study analyzed nested cohort data for 1238 patients from the Project of Ex Vivo Vein Graft Engineering via Transfection (PREVENT) III trial, the largest prospective randomized clinical trial evaluating the efficacy of edifoligide on outcomes in infrainguinal bypass for CLTI. The study identified 3 distinct clusters within the cohort, with 1-year CLTI-free survival rates of 82.3% for stage 1, 61.1% for stage 2, and 53.4% for stage 3. Meaning These findings suggest that, based on presenting characteristics, there are distinct clusters of patients with different 1-year CLTI-free survival; therefore, CLTI-free survival can be accurately and reproducibly quantified and may be used as a patient-centric outcome. Author affiliations and article information are listed at the end of this article. Open Access. This is an open access article distributed under the terms of the CC-BY License. JAMA Network Open. 2022;5(3):e223424. doi:10.1001/jamanetworkopen.2022.3424 (Reprinted) March 22, 2022 1/12 Downloaded From: https://jamanetwork.com/ on 03/25/2022 Abstract (continued)continued) CONCLUSIONS AND RELEVANCE The topic model cluster analysis in this cohort study identified 3 distinct stages within CLTI. Findings suggest that CLTI-free survival is an end point that can be accurately and reproducibly quantified and may be used as a patient-centric outcome. JAMA Network Open. 2022;5(3):e223424. doi:10.1001/jamanetworkopen.2022.3424


Introduction
Peripheral arterial disease is a common and debilitating manifestation of cardiovascular disease that impairs blood flow to the lower extremities. Chronic limb-threatening ischemia (CLTI) is the most severe manifestation of peripheral arterial disease and encompasses all patients with objective evidence of arterial insufficiency and symptoms of ischemic pain at rest and/or tissue loss (ulcerations, wounds, or gangrene). Required intensive medical therapy, revascularization, and/or wound management for patients with CLTI is complex, with highly variable patient responses and frequent reinterventions and even hospitalizations per year. Outcomes for patients with CLTI remain suboptimal despite evolving devices, medical therapeutics, and modern surgical techniques.
Comparative effectiveness research (CER) is essential to iteratively improve data-driven outcomes in CLTI, yet the overwhelming heterogeneity of treatments and patient presentations in CLTI stymies CER by preventing objective outcome comparisons. 1,2 Patients and physicians have limited highquality data to balance the competing mortality risk, limb threat, procedural invasiveness, hemodynamic efficacy, and longevity of a proposed procedure for a given patient with CLTI. 1,2 Although multiple researchers have developed baseline risk stratification schemes to enable CER in CLTI, none have been universally adopted. 2 All have modest predictive ability in CLTI, with model accuracy less than 72%. 1,2 No prior risk stratification model has been prospectively validated against large external data sets. End points of prior models are variable, often focusing on limb outcomes or survival. Few risk stratification schemes use modern computational advances, such as cluster analyses and/or machine learning to optimize model accuracy. 3,4 The Society for Vascular Surgery Wound, Ischemia, and Foot Infection (WIfI) score has been validated with regard to limbbased outcomes in retrospective cohorts. 5,6 Unfortunately, WIfI was designed to predict major amputation at only 1 year and is not understood with regard to mortality or other non-limb-specific composite outcomes.
Moreover, efficacy of infrainguinal bypass has been challenged over the past 2 decades because of the penetrance of endovascular devices and techniques. 7 Indeed, training patterns suggest that experience with infrainguinal bypass and in particular infrageniculate bypasses, may be waning. 8 No one is certain whether current outcomes reflect ideal infrainguinal bypass outcomes, and the comparisons between endovascular and infrainguinal bypass outcomes are potentially biased by this unmeasured confounding. Hence, it remains germane to model CLTI outcomes after infrainguinal bypass within a CLTI cohort in which the outcomes reflect what is truly achievable with infrainguinal bypass. Finally, only a small amount of literature focuses on patient-centric outcomes in CLTI. For example, outcomes analyses in CLTI focus frequently on amputation-free survival or patency.
Although important, these outcomes fail to capture patient-centric outcomes, such as relief of ischemic rest pain and wound healing.
In this context, we aim to recapitulate our novel technique to generate a machine learningbased risk stratification scheme for CLTI within the nested cohort data from the previously published Project of Ex Vivo Vein Graft Engineering via Transfection (PREVENT) III trial. 4,9 We also aim to evaluate the ability to accurately and reproducibly use CLTI-free survival as an end point in CLTI outcomes research.

Study Population
The PREVENT III randomized clinical trial comprised 1404 patients undergoing infrainguinal vein bypass for the treatment of ischemic rest pain or ischemic tissue loss. 9 Patients participating in the trial, which was monitored by a data and safety monitoring board, provided written informed consent on enrolling in the trial. In our cohort, we included all patients who had complete trial enrollment and outcome data. Original data were collected from January 1, 2001, to December 31, 2003, and were analyzed in September 2021. All patients had 1 year of follow-up. Nested cohort data from the PREVENT III trial were anonymized before data analysis. Because of the retrospective nature of the analysis and impracticality of obtaining patient consent years after the closure of the clinical trial, the institutional review board at the University of North Carolina waived consent. The study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline. 10

Outcomes
The primary outcome measured was 1-year CLTI-free survival, which is a composite end point that includes patients who are alive with remission of CLTI symptoms (wound healing and/or freedom from ischemic pain), no ipsilateral major limb amputation, and no recurrence of ipsilateral CLTI at 1 year. Secondary outcomes are the individual components of this composite end point. We considered 1-year CLTI-free survival as a binary outcome, and we classified patients as having 1-year CLTI-free survival if they were alive, were free of major lower-extremity amputation, were free of recurrence, and had healed their index wound at 1 year after the index procedure. Specifically, mortality, major limb amputation, and recurrence of CLTI and their respective dates were recorded in the case report forms and used directly from the trial data set. For patients who had an index wound, we used the follow-up wound examination records to determine whether the index wound was healed. If the index wound was recorded as healed before or at 1 year after the index procedure, the wound was considered healed. If it was recorded as unhealed at 1 year after the index procedure, the wound was considered unhealed.
For the wound grade of the WIfI score, we used data from the baseline examination of the index leg. Records indicated the presence of gangrene and/or ulcers for each toe, the forefoot, the hindfoot, or other location. Free text responses to wound locations coded as other were recoded to the appropriate corresponding category (toe[s], forefoot, hindfoot, or leg). Wound scores were assigned as follows: wound grade 0, no index wound; wound grade 1, an ulcer was reported on only 1 toe; wound grade 2, an ulcer was reported on multiple toes OR gangrene was reported for 1 or more toes or the forefoot; wound grade 3, an ulcer was reported on the hindfoot or leg OR gangrene was reported for the forefoot, hindfoot, or leg; and highest score, multiple wounds (eg, a toe ulcer [grade

Statistical Analysis
Demographic, comorbid, and CLTI-related characteristics for the cohort studied were described using medians (IQRs) for continuous variables and numbers (percentages) for categorical and binary variables. In accordance with the STROBE statement for reporting outcomes from observational studies, we do not include P value comparisons between the 2 groups in our description. 10 Using the strategy developed in McGinigle et al, 4 we used topic modeling from the natural language processing literature to cluster patients. Topic models are used to model the words in a corpus of documents to discover latent topics within those documents. In many cases, documents may have multiple topics; for example, movie reviews may be clustered by movie genre (eg, drama or horror) or by whether it was a positive review or not. Supervision is introduced to the model to guide topic formation so that topics are predictive of the desired response (eg, movie genre). In the context of our work, we considered our sample as a corpus of documents, with each individual in the sample as a document and the comorbid and CLTI-related features of each individual as the words in a document; supervision is provided by the outcome of interest, which in our case is CLTI-free survival.

JAMA Network Open | Surgery
To learn the latent topics or clusters in our sample, we specified a supervised latent Dirichlet allocation model, 11 which models responses, latent topics, documents, and words within documents.
The generative model supposes that documents are generated from a set of latent topics and those topics induce an unknown distribution over the vocabulary. Thus, each topic has words that are more likely or less likely to appear in a document that arose from that topic. Translating from text documents to patients with CLTI, this means that patients in a particular cluster are more likely to share common features (common vocabulary), for example, wound grade and history of myocardial infarction, than patients in a different cluster. In addition to looking for common patterns, the latent clusters are formed with supervision from the patient outcome of 1-year CLTI-free survival.
Specifically, CLTI-free survival enters the model as conditional on the proportion of features that are assigned to each cluster for the patient; we use a probit model, a type of generalized linear model for regression with binary outcomes. Given a fixed number of topics or clusters, we use the data to learn the probability that a patient belongs to each of the clusters and the distribution over the features within each cluster.
We slightly modified the model so that the regression for the response included demographic variables, such as sex and race, that were not included in the features used for topic formation, because demographic characteristics are likely to be associated with differential response owing to systemic disparities in health and health care but are not features on which we would want to use to characterize CLTI subtypes. To learn the model parameters, we implemented the collapsed Gibbs sampling approach described by Griffiths  clusters, the probabilities of patients belonging to a particular cluster were also computed. We considered patients to be in stage 1 if the probability of them being in stage 1 was higher than the probability of them being in stage 2 or stage 3 and similarly for stages 2 and 3. Clusters were characterized by the features with which they were most often associated and further characterized by analyzing the characteristics of the patients within them.
For this hypothesis-generating study, we emphasize that our model describes our cohort and provides insight into the feasibility of delineating limb-based and survival-based stages of CLTI.
Based on guidance from the American Statistical Association, we did not conduct null hypothesis significance tests to compare the latent stages discovered as one may in a formal confirmatory study. [13][14][15] Analyses were performed using R, version 4.0.4 (R Foundation for Statistical Computing) and Julia, version 1.6 (MIT Lincoln Laboratory).

Results
Of

Feature Differences in Each Stage Identified by Cluster Analysis of CLTI-Free Survival
Overall CLTI-free survival at 1 year was 60.0% (743 patients). We identified 3 clusters within the  significant differences between each stage are numerically quantified in Table 2 and graphically represented in Figure 1.
Comparisons to the overall population revealed that the 130 patients in stage 1 had the most variation from the overall pattern of patient features. Patients in stage 1 were more likely to have isolated femoropopliteal disease (with bypass targets more often to the above-or below-knee popliteal arteries), a history of smoking and hyperlipidemia, and smaller wounds. They were less likely to have diabetes, history of stroke or myocardial infarction, dialysis dependence, and grade 2 or 3 wounds.
Patients in stage 2 (n = 578) and stage 3 (n = 530) were more similar to the cohort pattern of features but had some key differences. Patients in stage 2 were more likely to be current smokers and have a more significant medical history, including renal disease, myocardial infarction, and stroke. Of interest, patients in stage 2 were also the most likely to have grade 3 wounds. Patients in stage 3 were more likely to have never smoked, but their peripheral arterial disease seems to be more driven by type 1 diabetes, kidney disease, and dialysis dependence. This finding correlates with known patterns of atherosclerotic disease, and patients in stage 3 were more likely than the cohort mean to undergo bypasses to tibial or pedal arteries. Patients in stage 3 were also more likely to present with grade 2 and 3 wounds. The prevalence of each feature by stage is graphically presented in Figure 2.

One-Year Mortality and Limb Loss by Stages Supervised by CLTI-Free Survival
For the entire cohort, 1-year mortality was 16.0%. When stratified by stage, mortality was as follows: 10.0% (13 of 130 observed deaths in stage 1) for stage 1, 13.5% (78 of 576 patients) for stage 2, and 20.2% (105 of 521 patients) for stage 3. We note a difference in denominators for stages 2 and 3 in the  Difference was calculated as stage minus study sample.

Discussion
In this cohort study, topic model cluster analysis showed the ability to discern 3 distinct clusters of patients within the nested cohort of the PREVENT III data. Chronic limb-threatening ischemia-free survival at 1 year by our modeled stages was 82.3% in stage 1; 61.1% in stage 2; and 53.4% in stage 3.
This method may ultimately prove superior to other methods of risk stratification in CLTI because of its ability to incorporate a wider variety of features compared with other recent risk stratification schemes, such as WIfI. Moreover, topic model cluster analysis can adjust as the prevalence and severity of risk factors change over time within a particular disease state. The malleability of topic model cluster analysis may prove more useful than prior methods of generating risk scores to evaluate modern outcome data and quality improvement projects.
In contrast to the findings of prior publications, 3,16 wound and ischemic severity were not necessarily associated with worse outcomes, which intuitively makes sense, because not all patients with ischemic rest pain will have superior outcomes compared with those with ulcers or gangrene, and patients with less systemic disease may have better wound healing even for grade 2 and 3 wounds. This lack of strict association with ischemic and wound severity shows the importance of using CLTI-free survival as a composite end point. We should disclose that perfect discrimination between each wound grade cannot be performed retrospectively. Still, our data also show the novel promise of precisely measuring CLTI-free survival as a clinical end point. Although CLTI-free survival has been recently used by other authors, 17,18 to our knowledge, our study represents the largest to accurately use CLTI-free survival as the primary end point. This is significant because CLTI-free survival succinctly encapsulates patient-centric outcomes, such as wound healing, and alleviation of ischemic rest pain. Further external validation of our hypothesis-generating topic cluster model will be required to confirm our findings.
Prior literature does not include the relief of ischemic rest pain as a clinical outcome measure.

Strengths and Limitations
There are several strengths to this study. The in-depth analysis of CLTI-free survival was long overdue and will contribute to a staging strategy to guide patients on treatment options based on outcomes more nuanced than just limb loss or death. Our novel machine learning and natural language processing methods represent a significant advance over prior literature regarding risk prediction and patient staging. This method that uses topic model cluster analysis can easily be applied to larger, more comprehensive data sets that also include specific vascular anatomy and target artery patency to further improve the predictive ability.
This research also has several limitations. A decade has passed since PREVENT III finished enrollment, and penetrance of wound care and optimal atherosclerotic medical care has changed over time, which may alter the expected outcomes of the clusters derived. Further delineation of specific distinguishing risk factors will be required also in analysis of other cohorts because the patients' features in stages 2 and 3 were similar to those of the overall cohort mean with the exception of smoking history and dialysis dependence, which are the current differentiators between those stages. In addition, we did not have precise hemodynamic and infectious data, which limited our ability to calculate a complete WIfI score. Still, we were able to incorporate the wound severity, which is the most significant component of the WIfI score among patients who have undergone revascularization. 3 The data were derived from a nested cohort of infrainguinal bypasses only; hence, we cannot extrapolate our results to populations undergoing endovascular and/or hybrid revascularizations.

Conclusions
This cohort study represents one of the largest and most comprehensive mathematically derived risk stratification schemes for CLTI. By topic model cluster analysis, 3 distinct stages of 1-year CLTI-free survival were identified. This study also showed that CLTI-free survival is an end point that can be accurately measured, which shows promise in improving patient-centric outcome assessments in CLTI.

ARTICLE INFORMATION
Accepted for Publication: January 31, 2022.