{'fraction_non_alphanumeric': 0.03480096973488699, 'fraction_numerical': 0.011574255102838821, 'mean_word_length': 5.16413593637021, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Who will Leave a Pediatric Weight Management Program and When? -- A machine learning approach for predicting attrition patterns

Childhood obesity is a major public health concern. Multidisciplinary pediatric weight management programs are considered standard treatment for children with obesity and severe obesity who are not able to be successfully managed in the primary care setting; however, high drop-out rates (referred to as attrition) are a major hurdle in delivering successful interventions. Predicting attrition patterns can help providers reduce the attrition rates. Previous work has mainly focused on finding static predictors of attrition using statistical analysis methods. In this study, we present a machine learning model to predict (a) the likelihood of attrition, and (b) the change in body-mass index (BMI) percentile of children, at different time points after joining a weight management program. We use a five-year dataset containing the information related to around 4,550 children that we have compiled using data from the Nemours Pediatric Weight Management program. Our models show strong prediction performance as determined by high AUROC scores across different tasks (average AUROC of 0.75 for predicting attrition, and 0.73 for predicting weight outcomes). Additionally, we report the top features predicting attrition and weight outcomes in a series of explanatory experiments.


Introduction
Childhood obesity is a major public health concern across the world. The prevalence of overweight and obesity among children and adolescents aged 2-19 has risen dramatically from 29% in 2000 to just over 32% in 2012 in the United States (Skinner and Skelton, 2014). Childhood obesity increases the risk of diabetes, cardiovascular disease, and cancer, predominantly as a result of a substantially greater risk of adult obesity (Ogden, Carroll, Kit and Flegal, 2014), but also causes significant morbidity in childhood. Multi-disciplinary clinical weight management programs (WMPs) are recommended for children with obesity who fail to improve with management in the primary care setting but these programs often require a moderate to high intervention dose delivered over an extended period (Ball, Sebastianski, Wijesundera, Keto-Lambert, Ho, Zenlea, Perez, Nobles and Skelton, 2021). Children and their families who attend more intervention sessions and remain enrolled in care for longer periods achieve the greatest improvements in weight and health (Wilfley, Saelens, Stein, Best, Kolko, Schechtman, Wallendorf, Welch, Perri and Epstein, 2017). However, due to a variety of reasons, such as dissatisfaction with the intervention progress or logistical issues with attending the programs, many families (as much as 80% (Ball et al., 2021)) discontinue attending WMPs prematurely (a phenomenon referred to as "attrition"). Besides not treating the disease effectively, a failed weight loss attempt may also lead to frustration, discouragement, and learned helplessness (Ponzo, Scumaci, Goitre, Beccuti, Benso, Belcastro, Crespi, De Michieli, Pellegrini, Scuntero, Marzola, Abbate-Daga, Ghigo, Broglio and Bo, 2020). Attrition can also be challenging for healthcare systems needing to ensure effective delivery of their services attrition prediction refers to predicting the time (number of weeks after the baseline visit) of the last visit, and the weight outcome prediction task refers to predicting the change in BMI percentile (BMI%) of patients at their last visit to the WMP (classified as a decrease in BMI% or not). Besides multi-task learning, our model also follows a "transfer learning" design, by training our model on various lengths of observation and prediction windows (Ball et al., 2021). Following the initial training of the model on these different settings, it is then fine-tuned (retrained) on a final target task to report the desired outcomes. Specifically, the main contributions of our study are: • We have collected one of the largest and most comprehensive datasets dedicated to study attrition. The dataset represents 4,550 children from four WMP sites within a large pediatric healthcare system, The dataset includes standard items from the EHR (e.g. medical diagnoses) and additional lifestyle and psychosocial factors collected by the WMPs in the EHR as part of routine clinical care. The dataset also includes temporal factors that may be important such as time between visits and BMI% trajectories.
• We present a deep model for predicting when attrition occurs and patients' BMI% change at that time. We use a multi-task and transfer learning approach for improving the performance of our model, facilitating its deployment in real settings without large training data.
• We compare our model against the commonly used logistic regression and a state of the art survival analysis method and show that it achieves better results in predicting attrition.
• We study the role of various input features in predicting attrition and successful BMI% change. This increases the interpretability of our model and makes our findings more actionable.

Related Work
Studying attrition is similar to studying other healthcare problems such as visit attendance and treatment adherence in clinical settings. Attendance prediction generally aims to predict whether a patient will show up for a scheduled appointment or not. Attendance prediction has been used to identify the important factors predicting visit attendance in various fields such as rehabilitation (Sabit, Griffiths, Watkins, Evans, Bolton, Shale and Lewis, 2008;Hayton, Clark, Olive, Browne, Galey, Knights, Staunton, Jones, Coombes and Wilson, 2013), psychiatric care (Mitchell and Selmes, 2007), and primary care (Giunta, Briatore, Baum, Luna, Waisman and de Quiros, 2013;Kheirkhah, Feng, Travis, Tavakoli-Tabasi and Sharafkhaneh, 2015). On the other hand, adherence prediction aims to predict whether a patient will be compliant with their treatment plan (e.g. taking prescribed medicines). Treatment and medication adherence have been studied for conditions such as tuberculosis (Killian, Wilder, Sharma, Choudhary, Dilkina and Tambe, 2019), heart failure (Son, Kim, Kim, Choi and Lee, 2010), and schizophrenia (Son et al., 2010). There are also other studies outside the healthcare domain that relate to our work. Among these related studies, one is "churn" prediction often used in economic domains to predict engagement patterns of individuals (such as customers and employees) to increase their retention. Churn prediction has been also well researched in the fields of banking (Ali and Arıtürk, 2014), video games (Kawale, Pal and Srivastava, 2009;Hadiji, Sifa, Drachen, Thurau, Kersting and Bauckhage, 2014), and telecommunication (Huang, Kechadi and Buckley, 2012).
While the studies discussed so far closely relate to the field of attrition, a major distinguishing aspect of attrition studies is the chronicity of the conditions being treated and the commitment to an intervention or treatment required for success. Examples of conditions where attrition patterns have been studied include mental health conditions (Linardon and Fuller-Tyszkiewicz, 2020), sleep disorders (Hebert, Vincent, Lewycky and Walsh, 2010), and addiction disorders (Murray, White, Varagunam, Godfrey, Khadjesari and McCambridge, 2013). Attrition from WMPs, in particular, has been studied in different settings, mainly using traditional methods such as linear and logistic regression (Jiandani et al., 2016;Altamura et al., 2018;Ponzo et al., 2020;Perna, Spadaccini, Riva, Allegrini, Edera, Faliva, Peroni, Naso, Nichetti, Gozzer et al., 2018). Linear and logistic regression assume that the relationship between the independent and the dependent variables are linear, which may not be correct. Using these standard methods, researchers have proposed various predictors of attrition, such as psychological (Altamura et al., 2018), (Jiandani et al., 2016), sociodemographic, and anthropometric (Ponzo et al., 2020) factors, and initial weight-loss (Perna et al., 2018). Among this family of studies, Batterham et al. (Batterham et al., 2017) were the only group who used shallow decision trees to predict attrition in dietary weight loss trials using demographic and early weight change characteristics. Their approach, however, did not use weight trajectories or other temporal patterns.

Dataset
Our dataset includes all children 0-21 years of age who visited one of the four WMPs of the Nemours Children's Health system between 2007 and 2021. The four WMPs serve patients in the US states of Delaware, Florida, Maryland, New Jersey, and Pennsylvania. For each patient, data from an internal dataset collected by the providers inside the WMPs were linked to EHR data, capturing their health records when interacting with the entire healthcare system. The internal dataset collected at the WMPs was specifically designed to capture important covariates generally missing in EHRs, such as psychosocial determinants of health and lifestyle behaviours (see Table 1). The EHR dataset was the Nemours portion of the large PEDSnet data repository and included rigorously validated EHR variables including medical conditions, anthropometrics, visits, and demographics Hispanic, 48% with Medicaid) with a mean BMI% (BMI percentile) of 98. For the BMIs, we use age-and sex-adjusted BMI% above the 95th percentile as defined by the Centers for Disease Control and Prevention (CDC) (Centers for Disease Control and Prevention, 2017). BMI% is recorded and fed to the model whenever available (for the baseline visit and the majority of the follow-up visits) and the sequence of BMI%s from baseline to the end time point of the particular prediction window is included as a dynamic variable. For each child, we included 18 features capturing demographic, psychosocial, lifestyle, anthropometric, medical comorbidity, and visit variables. Table 1 shows a list of these variables and their values. Our variables include two risk scores, capturing evidence-based lifestyle (diet, sleep, physical activity, sedentary activity, and hunger) and psychosocial (mental health and behavioural) factors. In our dataset, 28% of the children had only one WMP visit, and 15% had more than ten visits. Figure 1 shows the overall distributions of the number of visits and the duration of WMP attendance in months.
To bucketize the longitudinal EHR data, we combined visits over 15-days time periods. We examined medical diagnoses codes and medication tables, from the available EHR data. Any condition observed at least once during the time window was denoted by 1 in the new sequence, and the measurements were averaged over the time window. If there were no visits for a patient in a time window, the corresponding vector for that period was set to all zeros. We also excluded rare diagnosis codes (i.e., the codes that appeared in less than 2% of patients), which reduced the total number of diagnosis codes from 435 to 24. We used one-hot encoding for the categorical values and normalized the continuous values by performing a min-max scaling on all features.

Method
As discussed earlier, in this study we considered two related predictive tasks, i.e., attrition prediction and weight outcome prediction. We considered these two prediction tasks in a binary classification framework. In the attrition prediction task, patients who dropped out before the prediction window were considered positive cases. In the outcome prediction task, any decrease in the child's BMI% was considered a success. Accordingly, we defined the patients whose BMI% in the prediction window was lower than their BMI% at the time of their baseline visit to the WMP as negative cases. The rest of the patients (i.e., those whose BMI% remained the same or increased during the observation window) were considered as positive cases.
We used flexible observation and prediction windows for both tasks, where the start of the observation window was always fixed at the first WMP visit with a rolling end considered for the end of the window. The start of the prediction window was considered as the end of the observation window with a flexible end. We note that this type of flexibility in defining observation and prediction windows makes our models more practical in clinical settings, where a provider needs to know which child will leave at what time following the start of the program.
For implementing the two prediction tasks, we propose a deep neural network architecture, specifically designed to address our problem's needs. In order to fully utilize the static (e.g., demographics) and temporal features (e.g., measurements) in our data, our architecture is designed with the following four components as shown in Figure 2. Two initial components are shared between the two tasks of attrition and outcome predictions and the other two are task-specific com- ponents. The first component is a two-layer fully-connected network for processing the static features. The second is a two-layer bi-directional long short-term memory (Bi-LSTM) network for processing the temporal features. Bi-LSTM structures are similar to the common LSTMs, but they consist of two LSTMs units: one taking the input in a forward direction (e.g., left to right), and the other in a backwards direction (e.g., right to left), thus improving the available context for the model (Schuster and Paliwal, 1997). The third component is a three-layer fully connected network for combining the extracted feature vectors from the first two parts and predicting the attrition time. Finally, the fourth component is a three-layer fully-connected network for combining the extracted feature vectors from the first two parts and predicting the outcome. We used dropout and batch normalization layers after all of the layers mentioned above. For training the model, we used binary cross-entropy as the loss function, defined as: where, N is the total number of samples, is the ground truth for the th sample, is the probability of belonging to the positive class , and (1− ) is the probability of belonging to the negative class. In the attrition task, we train a model to discriminate between the patients who dropped out and patients who stayed in the WMP. In the outcome task, we train a model to discriminate between patients who successfully decreased their BMI% and those who had no change or a BMI% increase.
To improve the overall performance of our models and accommodate the relatively small size of our training data, we use a "multi-task learning" approach for designing our model. In our design, we use a hard parameter sharing approach, which is a common multi-task learning method (Caruana, 1993), by sharing the first two components between the two tasks. This way, sharing the parameters of the two initial parts of the model can improve the performance for both predictions tasks. Additionally, following a "transfer learning" theme, we iteratively train our model on all sliding observation and prediction windows, and then fine-tune the model for each specific window settings. Figure 3 shows the training process and the way that the four components are involved in our multi-task and transfer learning themes. Additionally, Algorithm 1 presents a high-level pseudocode of our customized model training process. In this algorithm, each model is trained using the data from a specific observation and prediction window, initialized by the weights from the previous window settings. The procedure in this algorithm receives the input data ( ), labels for the attrition and outcome tasks ( and ), and the list of observation and prediction windows. It returns a list of fine-tuned models.  Figure 3: Proposed training process using the multi-task and transfer learning themes.
, , , and are the weights of the static, attrition, temporal, and outcome sub-networks, respectively. We extract the features and the corresponding labels for each patient based on the rolling observation and prediction windows and feed them to the network. After the pretraining of a general model, we then initialize the weights of the specialized models with the weights from the general model. For each specific observation and prediction window setting, the model is then fine-tuned using only the relevant samples.
shows the th fine-tuned configuration.
The model presented in this paper was implemented using Keras (Chollet et al., 2015) inside the TensorFlow (Abadi, Barham, Chen, Chen, Davis, Dean, Devin, Ghemawat, Irving, Isard et al., 2016) framework. Our code is publicly available on GitHub 1 . Add M to trained models list 13: return Trained models list

Experiments and Discussion
To measure the prediction performance of our model, we report accuracy, precision, recall, area under the receiver operating characteristic (AUROC), area under the precisionrecall curve (AUPRC), and baseline AUPRC. The last measure (baseline AUPRC) is the proportion of positive exam-1 https://github.com/healthylaife/WM_attrition ples in our data. The performance of the proposed model in the outcome and attrition prediction tasks for a series of observation and prediction windows (observation window = 1, 2, 3, 4, 6, and 9 months; and prediction window = 1.5, 3, 4.5, 6, 9, 13.5 months) are shown in Table 2 and Table 3. This specific set of observations and windows were selected based on prediction windows used in prior studies (Moran et al., 2019;Jiandani et al., 2016) and based on the distribution of our data. The results show that the performance of the models has a direct relationship with the length of the observation window, i.e., as we expand the observation window through different tasks, the model's performance also increases. We compare our model against two separate baselines. The first is a general logistic regression (LR) method, which is a frequently used method in the literature (Shipe, Deppen, Farjah and Grogan, 2019). Similar to the common practice in the field, we aggregate the temporal variables by calculating the average values during a corresponding period. We train an LR model with L2 penalty and BFGS solver using the scikit-learn library (Pedregosa, Varoquaux, Gramfort, Michel, Thirion, Grisel, Blondel, Prettenhofer, Weiss, Dubourg et al., 2011) in Python.
For the second baseline, we use another popular method for studying similar problems in biomedical domains, which takes a survival analysis approach. Survival analysis (timeto-event analysis) is widely used in many engineering, economics, and medicine (Clark, Bradburn, Love and Altman, 2003) applications to estimate the expected duration until a desired event occurs (e.g., drop out or death). We use a state of the art method for studying survival analysis, called Dynamic DeepHit (Lee, Yoon and Schaar, 2020), which presents a deep neural network to learn the distribution of survival Table 2 Results for the attrition prediction task using our proposed method. Observation and prediction windows are shown in months.  Table 3 Results for the outcome prediction task using our proposed method. Observation and prediction windows are shown in months. times. Dynamic DeepHit utilizes the available temporal data to issue dynamically updated survival predictions. This survival analysis method only fits our attrition prediction task (and not the weight outcome prediction task). In an ablation analysis, we also study the role of the multi-task and transfer learning themes used in our design. To this end, we report the results of our method without (a) multi-task learning implementation (i.e., the two shared sub-networks and only one of the latter sub-networks for each task), and (b) transfer learning implementation (i.e., fine-tuning the models on the corresponding data to each observation-prediction window setting without pretraining). Figure 4 shows how the results obtained from our method compares to the two baselines and the ablated version of our model. Moreover, to study to what degree each input feature contributes to predicting the final outputs in each task, we used the popular Shapley additive explanations (SHAP) method implemented by the SHAP toolbox (Lundberg and Lee, 2017b,a). Inspired by the Shapley values, this toolbox is a unified framework that assigns each input feature an importance value indicating its importance in the prediction task. A SHAP value indicates the degree to which a feature contributes to pushing the output from the base value (average model output) to the actual predicted value, so the higher value means the higher importance and contribution toward the outcome of the model. Here, we report the importance values (as indicated by SHAP) for the top 5 features predicting each outcome and for each prediction/observation window. These are shown in tables 4 and 5. Not surprisingly, the top features did vary some by the outcome and prediction/observation window, which demonstrates the importance of targeting different risk factors at different points in treatment. However, age, the average time between the visits, and patient's BMI% trajectory were important features for determining both attrition and weight outcomes, which is consistent with previ-ous studies (Batterham et al., 2017;Batterham, Tapsell and Charlton, 2016). Specifically, a patients' early weight loss is predictive of overall weight loss progress and the patients who have success with early weight loss seem to have a lower risk of attrition. Similar to our study, other studies (Jiandani et al., 2016;Batterham et al., 2016) have also found that age is an important predictor of both attrition and weight outcomes, with children of younger ages having more success in WMPs. We have also found that the average time between WMP visits is a primary predictor for both tasks, with worse outcomes and higher attrition rates for patients who had a prolonged time between visits. This is consistent with the literature and current guidelines recommending a certain number of contact hours and frequency of contact for success in WMPs. Finally, sociodemographics like race and ethnicity and sex, as well as food security and insurance status, are important predictors of both attrition and weight outcomes, consistent with previous studies demonstrating inequities in health outcomes between subgroups (Martin and Ferris, 2007). Interestingly, other factors like medical diagnoses, medications, and lifestyle scores were less predictive of attrition and weight outcomes, which highlights the importance of identifying and supporting key groups based on sociodemographics, as well as ensuring frequent visits and early success with weight outcomes during treatment regardless of a patient's underlying conditions or lifestyle behaviors. We note that while we report the average SHAP values across all of the samples (children), our model is most helpful when individual children are considered separately, and a provider can see the predictors that can be targeted for a particular child to prevent attrition or increase success with weight outcomes.

Observation Predication Precision
The current study is limited in several ways. First, our dataset only includes the patients from one healthcare system. Still, our dataset is larger than similar ones used to  study attrition and spans four geographically distinct sites in the Mid-Atlantic (Delaware, Pennsylvania, Maryland, New Jersey) and Southern (Florida) regions of the US. Additionally, our approach relies on discretizing future time into twoweek windows. Considering attrition prediction as a regression task was an alternative natural choice. In our experiments, we have noticed that formulating our problem as a classification task yields better results. Additionally, as most follow-up visits are not scheduled in shorter than two-weeks intervals, one can still use our approach for continuous (any time in future) predictions. While our study focuses primarily on attrition from pediatric WMPs, our method should be applicable to adult obesity WMPs and other comparable problems such as mental health and addiction recovery programs. As part of our future work, we plan to expand our model by including additional information from children's historical (before joining a WMP) records and also by including new data from other pediatric health systems. On the technical side, we plan to use a transformer-based architecture (Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin, 2017), which offers state of the art sequence-to-sequence predictive modelling to check whether it improves our model's performance further. Moreover, we aim to explicitly identify the temporal patterns (such as distinct shapes of the body-weight trajectory) that can predict (or stratify) attrition or outcome patterns.

Conclusion
In this study, we presented a deep neural network that includes Bi-LSTM elements for predicting attrition patterns and weight outcomes in pediatric weight management programs. We trained our model with a large electronic health records dataset collected from the Nemours Children's Health. Our model showed strong prediction performance as determined by AUROC scores across different tasks and superiority against other baseline methods, including logistic regression and a survival analysis method. Finally, we explored the importance of individual variables in predicting attrition and weight outcomes and found that sociodemographic variables as well as early weight trajectory and visit timing were important across both outcomes.