{'fraction_non_alphanumeric': 0.029009421408420718, 'fraction_numerical': 0.01708628306807899, 'mean_word_length': 5.212962962962963, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Identifying patients at increased risk of hypoglycaemia in primary care: Development of a machine learning‐based screening tool

Abstract Introduction In primary care, identifying patients with type 2 diabetes (T2D) who are at increased risk of hypoglycaemia is important for the prevention of hypoglycaemic events. We aimed to develop a screening tool based on machine learning to identify such patients using routinely available demographic and medication data. Methods We used a cohort study design and the Groningen Initiative to ANalyse Type 2 diabetes Treatment (GIANTT) medical record database to develop models for hypoglycaemia risk. The first hypoglycaemic event in the observation period (2007–2013) was the outcome. Demographic and medication data were used as predictor variables to train machine learning models. The performance of the models was compared with a model using additional clinical data using fivefold cross validation with the area under the receiver operator characteristic curve (AUC) as a metric. Results We included 13,876 T2D patients. The best performing model including only demographic and medication data was logistic regression with least absolute shrinkage and selection operator, with an AUC of 0.71. Ten variables were included (odds ratio): male gender (0.997), age (0.990), total drug count (1.012), glucose‐lowering drug count (1.039), sulfonylurea use (1.62), insulin use (1.769), pre‐mixed insulin use (1.109), insulin count (1.827), insulin duration (1.193), and antidepressant use (1.05). The proposed model obtained a similar performance to the model using additional clinical data. Conclusion Using demographic and medication data, a model for identifying patients at increased risk of hypoglycaemia was developed using machine learning. This model can be used as a tool in primary care to screen for patients with T2D who may need additional attention to prevent or reduce hypoglycaemic events.


| INTRODUCTION
Strict glucose control in patients with type 2 diabetes (T2D) is essential for preventing long-term micro-and macro-vascular complications, such as retinopathy, neuropathy, and cerebrovascular, kidney and heart disease. [1][2][3][4] However, strict control of T2D with insulin and insulin secretagogues, especially in frail and older patients, increases the risk of hypoglycaemia. [5][6][7] Commonly, three levels of hypoglycaemia are distinguished. Glucose levels between 3.9 mmol/L and 3.0 mmol/L are defined as mild hypoglycaemia and are considered an alert value. 8 Glucose levels below 3.0 mmol/L are defined as moderate hypoglycaemia and are considered clinically important events, whereas severe hypoglycaemia is defined by a mental or physical impairment that requires external assistance. Hypoglycaemia is not only associated with reduced quality of life and higher healthcare costs but also hinders strict glucose control. [9][10][11][12] Severe cases of hypoglycaemia can lead to unconsciousness, hospitalization, and even brain death. Hypoglycaemia during hospital admissions has been associated with prolonged hospital stay and increased mortality. 13 In addition to the mortality directly caused by hypoglycaemic events, several studies have shown an association of hypoglycaemia with all-cause and cardiovascular mortality. [14][15][16] In T2D patients, the risk of hypoglycaemia strongly varies among patients. Not all T2D patients require insulin or insulin secretagogues to control their glucose level. 17 Moreover, differences in co-medication can influence the risk of hypoglycaemia.
For instance, non-selective beta-blockers reduce the effects of epinephrine, which in turn, increases hypoglycaemia unawareness, 18 and selective serotonin reuptake inhibitors can increase sensitivity to insulin. 19 In addition to differences in prescribed medications, differences in comorbidities can also influence the risk of hypoglycaemia. Dementia, for instance, reduces patients' ability to manage their glucose-lowering medications and increases the risk of medication errors, 20,21 and depression can lead to poor selfcare and self-monitoring. 22 Furthermore, decreased renal and liver functions can diminish the clearance of both endogenous and exogenous insulin. 18,23 Several models have been developed to identify patients at high risk of hypoglycaemia. [24][25][26][27][28][29][30] Most of these models have been developed using a mix of type 1 diabetes patients and T2D patients, focus on selecting high risk patients in inpatient settings, and often include previous hypoglycaemic events to predict future events. Most hypoglycaemic events occur outside the hospital setting and therefore need to be managed by primary care. 31,32 In primary care, stratifying patients based on hypoglycaemia risk can provide an approach in which treatment, glycaemic goals and education are tailored to the individual, which ultimately helps to reduce hypoglycaemia. 33 Previous research has shown that pharmacy-led interventions can be effective in reducing the number of hypoglycaemic events in T2D patients. 34,35 Using routinely available pharmacy data to screen for patients at increased risk of hypoglycaemia can improve the effectiveness and efficiency of these interventions. Generally, pharmacy data include information about patients' medication history and all drugs dispensed during a specific period. Conventional approaches to using such information in risk models include creating simple measures of medication use at the substance or therapeutic class level. 36 Recent developments in machine learning provide statistical tools to include multiple characterizations of medication use in one model, for example, 'current insulin use', 'insulin count', 'insulin type', and 'insulin use duration'. In this way, more information can be utilized for risk modelling.
In this study, we aimed to develop a screening tool to identify T2D patients at increased risk of hypoglycaemic events based on machine learning using demographic and medication data available in community pharmacies. Additionally, we investigated whether the performance of the model could be improved by adding electronic clinical data that are available in general practice. We also compared the results of these models with results from traditional regression models.

| Study design
We conducted a cohort study using demographic and medication data to predict the occurrence of hypoglycaemia in T2D patients. The performance of this model was compared to that of a model that additionally included clinical data. Data from the Groningen Initiative to ANalyse Type 2 Diabetes Treatment (GIANTT) were used (www. giantt.nl). This database includes data from a longitudinal dynamic cohort of more than 60,000 patients with a general practice (GP)confirmed diagnosis of T2D. The data were extracted from the electronic medical records of 180 GPs in the northern parts of the Netherlands. 37 At the end of our study period, approximately 80% of all the GPs in this region contributed to the GIANTT database. The database contains patients' demographic data, data on prescribed medications (similar to medication data in a pharmacy information system), medical history based on the International Classification of Primary Care (ICPC) 38 and routinely available clinical measurements, such as blood pressure and low haemoglobin A1c (HbA1c) levels.

| Outcome and predictor variables
The outcome of this study was the first hypoglycaemic event during the observation period between January 2007 and January 2014. Hypoglycaemic events were defined by the ICPC code for hypoglycaemia (T87) or by a recorded glucose measurement below 3.9 mmol/L. We therefore included all the levels of hypoglycaemia that were recorded in the medical records by GPs since mild events can also be relevant for our screening tool. Recorded hypoglycaemic events in free text parts of the medical record were used to enrich the T87 codes in the GIANTT database. The date of the first hypoglycaemic event was used as the index date. Patients without any hypoglycaemic events were assigned a random index date in the observation period. When assigning these random dates, the distribution of the index dates of the event group was taken into account to correct for temporal influences. Based on the literature, potential predictor variables related to demographics, medication, and clinical information were selected. An overview of the selected variables and their definitions can be found in Appendix SI.
Any prescription in the 4 months prior to the index date was used to determine current medication use, the latest measured value within 1 year prior to the index date was used for most clinical measurements and active ICPC codes at the time of the index date were used for comorbidities. For the medication data, potential predictor variables included not only the current use of medication but also other aspects of medication use, including duration of use, counts of medication classes, and potential drug-drug interactions.

| Study population
This study included T2D patients who were 18 years or older and were treated for diabetes in a primary care setting. We excluded patients registered with a GP with poor documentation of hypoglycaemia, as demonstrated by a lack of recordings for hypoglycaemia (T87) for any of their T2D patients. The following patient exclusion criteria were applied: (A) included in the database for less than 1 year prior to their index date; (B) no prescription of a glucoselowering medication 1 year prior to the index date; (C) no T2D diagnosis before or within 6 months after the index date; and (D) missing all of the following measurements 1 year prior to the index date: estimated globular filtration rate (eGFR), creatinine, albumin to creatinine ratio, HbA1c, weight, body mass index, high density lipoprotein, low density lipoprotein, total cholesterol, systolic blood pressure, and diastolic blood pressure.
Due to the imbalanced dataset (N = 2,523 and 11,344 with and without a hypoglycaemic event, respectively), which could severely bias the performance of machine learning algorithms, we used a sample-size equalization method to balance the data. In this method, we randomly divided the patients without a hypoglycaemic event into four equal subdatasets, corresponding to the number of patients in the hypoglycaemic group, as shown in Figure 1 (11,344/2,523 = 4 subdatasets). Model training was performed using these balanced datasets.

| Analysis
We first evaluated the prediction performance of all the individual variables and evaluated the performance of traditional logistic regression using either all the demographic and medication data or all the demographic, medication and clinical data. Next, to develop the screening tool, we trained several machine learning algorithms on either demographic and medication data or demographic, medication, and clinical data. We evaluated the performance of the following algorithms: logistic regression with backwards selection, ridge logistic regression, elastic net logistic regression (α = 0.1, 0.2, 0.3, 0.4 0.5, 0.6, 0.7, 0.8, or 0.9), least absolute shrinkage and selection operator (LASSO) logistic regression and random forest (RF). These algorithms were used because they are relatively simple to interpret, easy to implement in practice, and, in the cases of elastic net, LASSO and RF are suitable for dealing with multidimensional data. 39,40 Missing clinical measurement data were imputed using the K-nearest neighbour (k = 5) multiple imputation method (10-fold). The list of the predictor variables and the percentage missing for each variable are given in Appendix SI. We performed fivefold cross-validation (80% training data and 20% testing data) to evaluate the performance of the models using area under the curve (AUC) as a metric. The best performing model without clinical data was compared with the best performing model including clinical data. Stata 15.0 was used for data cleaning, logistic regression, and RF. R version 3.5.1 was used with the glmnet package for LASSO, elastic net, and ridge logistic regression. 41 For RF, out of the bag error was used to find the optimal number of trees and the optimal number of variables considered at each node. No maximum depth was set, and the minimum number of observations per leaf was set to one.
We performed fivefold cross-validation to identify optimal values for the lambda plus one standard error for LASSO, elastic net, and ridge logistic regression. Second, we performed fivefold cross-validation to identify the optimal α for the elastic net between 0.1 and 0.9 with steps of 0.1. Third, we performed fivefold cross-validation to select the balanced subdataset and the imputed dataset that resulted in the best performance. The predictor variables were standardized for the elastic net, ridge, and LASSO logistic regression analyses. A heatmap was created to show the importance of the variables used in the best performing model.

| Ethics statement
Based on the research code of conduct in the Netherlands, research using anonymous medical record data requires no ethics committee approval.

| RESULTS
We extracted 40,124 patients from the GIANTT database. After exclusion, 13,876 patients remained, 19.2% of whom had at least one hypoglycaemic event during the observation period (Appendix SII).
The subdatasets contained either 5201 or 5202 patients. Table 1 shows that patients with and without a hypoglycaemic event were similar in age and clinical measurements, but patients with an event used insulin more often and had a longer diabetes duration.

| Models with demographic and medication data
Using the demographic and medication data, the best predicting individual variable was 'insulin use duration', with an AUC of 0.64 CRUTZEN ET AL.

| Main findings
A model including 10 demographic and medication variables based on machine learning showed an acceptable performance to screen for an F I G U R E 1 Schematic overview of the sample-size equalization method and fivefold cross-validation to evaluate and compare the performance of the different machine learning models. To balance the data, the non-hypoglycaemia patients were divided in four equal groups, which were each matched with the hypoglycaemia patients. This was followed by fivefold cross-validation in each of the four subdatasets to determine which machine learning method in which subdataset resulted in the best performing model using area under the curve (AUC) as metric T A B L E 1 Characteristics of patients with and without a hypoglycaemicevent

| Comparison with previous research
Several models have been developed for predicting hypoglycaemia. [24][25][26][27][28][29][30] However, these models mainly focus on the inpatient setting or do not make a distinction between type 1 and T2D. Our model is intended for primary care using demographic and medication data that are widely available to screen for T2D patients with an increased risk of hypoglycaemia. By including only AUCs than ours. [28][29][30]43 The higher performance of these models may be due to the availability of richer data in clinical trials and inpatient settings in comparison to data that are routinely available in outpatient settings. For example, daily glucose measurements may be available for diabetes patients who are admitted to a hospital but not for T2D patients in primary care. More importantly, all of the better performing models included prior hypoglycaemic events as a variable, which was one of the, if not the most, important predictor in these models. Since we aimed to screen for patients at increased risk of a first hypoglycaemic event, we did not use prior events as a predictor. Predicting the first hypoglycaemic event is more difficult, but it is essential to identify at-risk patients who are not already known to healthcare professionals.
Many of the variables in our models are known risk factors for hypoglycaemia, such as using different types of insulin or using pre-mixed insulin. Previous research has shown that a longer diabetes duration is predictive of hypoglycaemia. 30 In our models, as in previous research, a longer duration of insulin use was associated with a higher risk of hypoglycaemia. 44 When clinical data were added to that model, diabetes duration-in addition to a longer insulin use duration-was associated with a lower risk. Surprisingly, lower age contributed to an increased risk of hypoglycaemia in our models, whereas in other models, higher age was predictive of hypoglycaemic events. 26,[28][29][30] This difference might be due to differences in the severity of the hypoglycaemia events used as outcomes. Previous studies mainly used severe hypoglycaemic events as the outcome, while we included any events regardless of severity. It has been found that younger patients may report more hypoglycaemic events, in general, but not more severe hypoglycaemic events. 45 This finding might be due to the increased hypo-unawareness in older patients. 46 The inclusion of antidepressant use as a predictor could be explained T A B L E 2 Odds ratios of the predictor variables for the best performing models (LASSO) using only demographic and medication data or additional clinical data

Predictors Odds ratio
Demographic and medication data

| Strengths and limitations
By using machine learning to develop our models, we were able to include several related variables, which allowed us to make better use of the information available when only using medication data.
Particularly, considering more information about insulin than just current use improved the performance of the model. By using LASSO to select variables, a simpler model could be obtained.
LASSO logistic regression is able to select predictors without having to rely on p-values, which are highly dependent on power.
Another strength is the use of cross-validation, which provides a less biased estimation of the performance of models. The use of a large database, which consists of routinely available primary care data, has two advantages. First, using routinely available data will mimic daily practice more closely, resulting in a more accurate estimation of the performance of the prediction models. Second, by using these data, only predictors that are commonly available and well documented in primary care are included in the prediction model.

| CONCLUSION
We developed a model for identifying patients at increased risk of hypoglycaemia in primary care using demographic and medication data based on LASSO, which can be used as a screening tool. This model had an acceptable performance, outperformed individual predictor variables and performed similarly to a model that used additional clinical data.