{'fraction_non_alphanumeric': 0.03695687029020362, 'fraction_numerical': 0.029140612473945806, 'mean_word_length': 4.871734525770769, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Prediction of low pulse oxygen saturation in COVID-19 using remote monitoring post hospital discharge

Background Monitoring systems have been developed during the COVID-19 pandemic enabling clinicians to remotely monitor physiological measures including pulse oxygen saturation (SpO2), heart rate (HR), and breathlessness in patients after discharge from hospital. These data may be leveraged to understand how symptoms vary over time in COVID-19 patients. There is also potential to use remote monitoring systems to predict clinical deterioration allowing early identification of patients in need of intervention. Methods A remote monitoring system was used to monitor 209 patients diagnosed with COVID-19 in the period following hospital discharge. This system consisted of a patient-facing app paired with a Bluetooth-enabled pulse oximeter (measuring SpO2 and HR) linked to a secure portal where data were available for clinical review. Breathlessness score was entered manually to the app. Clinical teams were alerted automatically when SpO2 < 94 %. In this study, data recorded during the initial ten days of monitoring were retrospectively examined, and a random forest model was developed to predict SpO2 < 94 % on a given day using SpO2 and HR data from the two previous days and day of discharge. Results Over the 10-day monitoring period, mean SpO2 and HR increased significantly, while breathlessness decreased. The coefficient of variation in SpO2, HR and breathlessness also decreased over the monitoring period. The model predicted SpO2 alerts (SpO2 < 94 %) with a mean cross-validated. sensitivity of 66 ± 18.57 %, specificity of 88.31 ± 10.97 % and area under the receiver operating characteristic of 0.80 ± 0.11. Patient age and sex were not significantly associated with the occurrence of asymptomatic SpO2 alerts. Conclusion Results indicate that SpO2 alerts (SpO2 < 94 %) on a given day can be predicted using SpO2 and heart rate data captured on the two preceding days via remote monitoring. The methods presented may help early identification of patients with COVID-19 at risk of clinical deterioration using remote monitoring.


Introduction
The COVID-19 pandemic has resulted in significant morbidity and mortality, placing a major strain on healthcare systems worldwide. Symptoms of COVID-19 can persist for weeks or months, and a long lasting sequelae of COVID-19 infection, or 'long covid', is now a recognised diagnosis [1][2][3]. While initial COVID-19 symptoms are frequently mild, clinical deterioration to severe disease is common in certain populations and predicting those who will deteriorate can be difficult [4]. In patients with COVID-19 recently discharged from hospital, remote monitoring can enable early recognition of patient deterioration and facilitate reassessment and readmission if necessary [5][6][7]. Remote monitoring may also allow identification of patients experiencing low oxygen saturation levels in the absence of signs of respiratory distress [8][9][10][11].
Previously reported systems for remote monitoring of patients with COVID-19 following hospital discharge have used mobile apps [6,7] and daily surveys [5,12] to monitor oxygen saturation (SpO 2 ), heart rate (HR), body temperature, and self-reported symptoms [5]. To enable early detection of worsening clinical status, the monitoring systems sent automatic alerts to the clinical team if SpO 2 , HR, temperature or breathlessness score passed defined thresholds [5][6][7]. These systems enabled early identification of patients who required additional assessment or intervention, supporting hospital discharge during a period of intense demand on hospital capacity, and demonstrated high levels of patient adherence.
Clinical remote monitoring systems, however, have not yet been harnessed to investigate how symptoms and vital sign measures change over time in patients with COVID-19, or to develop methods to predict clinical deterioration which may enable more efficient remote monitoring of large patient groups. Machine learning models have been used to predict clinical deterioration in patients hospitalised with COVID-19 based on clinical and laboratory data [13][14][15][16][17][18][19][20], and hospitalisation and illness in individuals with COVID-19 using wearable sensors [21,22]. Models using clinical data from patients hospitalised with COVID-19 have also been used to predict intensive care unit admission [13,14], in-hospital mortality [13][14][15], clinical deterioration defined as either intubation, intensive care unit admission, in-hospital mortality, or a high National Early Warning Score [16][17][18][19][20]. Low SpO 2 and elevated respiration rate have also been reported as risk factors for in-hospital death using multivariate regression [15,23]. Prior to hospitalisation, models have been reported to predict COVID-19 infection [21], hospitalisation [22] and sickness on a given day [22] in individuals with COVID-19 based on respiration and HR measures using large datasets obtained from consumer wrist-worn devices. A method to predict clinical deterioration in patients with COVID-19 following discharge from hospital using objective remote monitoring data, such as SpO 2 and HR, has not yet been reported.
In this study, SpO 2 , HR and breathlessness data recorded using a remote monitoring system in 209 patients with COVID-19 following discharge from hospital were retrospectively examined. A predictive model was developed to identify patients at risk of experiencing low SpO 2 (<94 %) requiring assessment and possible readmission. The associations between age and sex, and symptom presentation when SpO 2 was<94 % were also investigated.

Remote monitoring system
During 2020, a remote monitoring system was used by the Health Service Executive in Ireland as an early discharge pathway for patients with COVID-19 [6]. On discharge from hospital, patients with COVID-19 were provided with a Bluetooth Smart Pulse Oximeter (NoninConnect 3230, Nonin Medical Inc., Plymouth, MN, USA) linked to a mobile application to monitor SpO 2 , HR and self-reported breathlessness (patientMpower ltd, Dublin, Ireland). Recorded data were encrypted and sent to a secure cloud database accessible only to the hospital-based COVID-19 monitoring teams. The mobile application sent a prompt to check SpO 2 at rest, four times daily for 14 days after hospital discharge, and patients could enter additional data/measurements at will. SpO 2 and HR were measured by the pulse oximeter, and data were automatically captured by the mobile application. A prompt also asked if the patient felt more breathless than usual. If the patient selected "yes", they were asked to rate their breathlessness using a visual analogue scale, ranging from 1 (no symptoms) to 10 (worst ever). A clinical alert was generated if SpO 2 was < 94 %, with an SMS text sent to the hospital monitoring team for clinical follow-up. Age and sex were optional entries in the monitoring system.

Dataset
Data recorded by patients with COVID-19 who used the remote monitoring system following discharge from six Irish hospitals in the period 1st April -30th June 2020 were examined retrospectively in this study. During this period, 209 patients (aged > 18 years) agreed that their de-identified data could be used for research purposes (via an 'opt in' statement in the mobile app), and their de-identified data were shared with the researchers. Ethical approval was obtained from the Human Research Ethics Committee at University College Dublin and a consent declaration for the retrospective analysis of these data was obtained from the Health Research Consent Declaration Committee.

Data analysis
SpO 2 , HR and breathlessness data from the first ten days after hospital discharge were examined. For each patient, the mean and coefficient of variation (CV) of all SpO 2 , heart rate and breathlessness measures were calculated for each day. The minimum SpO 2 , and the maximum HR and breathlessness scores were also calculated daily for each patient. Linear mixed-effects models were used to examine the effect of time (days since discharge) on these nine variables, with a random intercept for each patient. The threshold for significance was p < 0.005, adjusted using Bonferroni's method accounting for multiple comparisons [24].
For every patient, each day was categorised as an alert day (SpO 2 < 94 %), or a non-alert day (SpO 2 ≥ 94 %). An SpO 2 threshold of 94 % was chosen for consistency with the clinical remote monitoring system used to record the data [6], and with the threshold for clinical alerts used by other remote monitoring systems for patients with COVID-19 who have been discharged from hospital [5,7]. However, it should be noted that in hospitalised patients with COVID-19, lower SpO 2 thresholds would be more appropriate to define clinical deterioration [25]. Alert days were further categorised as symptomatic or asymptomatic, using heart rate and breathlessness scores recorded while the patient was resting. On an alert day, if breathlessness (breathlessness score > 1) or an elevated heart rate (>100 bpm) were recorded, this day was considered a symptomatic alert day. If breathlessness was not reported (breathlessness score = 0) and an elevated heart rate was not detected (<=100 bpm) on a day when SpO 2 < 94 % was recorded, the day was considered an asymptomatic alert day.
To investigate how age and sex are associated with the occurrence of asymptomatic alerts, three patient groups were examined: patients who experienced at least one asymptomatic alert day (Asym), patients who experienced symptomatic alert days, and did not experience any asymptomatic alert days (Sym), and patients who did not experience any alert days during the ten days (NoAlert). Age and biological sex were compared across patient groups using linear mixed-effect models, with a random effect for patients, and fixed effects for patient group (Asym/ Sym/NoAlert), age and sex. Differences in overall mean HR and overall mean SpO 2 between patient groups were also examined, with age and sex considered fixed effects with a random subject-specific intercept. The threshold for significance was p < 0.015, adjusted using Bonferroni's method accounting for the four models developed [24].

Predictive model development
A random forest classifier was developed to predict SpO 2 alerts (SpO 2 < 94 %) on a given day (Day A ) in the period Day 3-10 (Day 1 was the day of hospital discharge), using SpO 2 and HR data from the discharge day and the two days preceding Day A (Day A-1 , Day A-2 ), and validated internally using tenfold cross-validation.
For the 196 patients who used the monitoring system for at least 3 days, 22 features were extracted and considered during sequential feature selection. For discharge day, the day before (Day A-1 ) and two days before (Day A-2 ) the day of interest (Day A , where A = 3-10): mean, CV and minimum SpO 2 , mean, CV and maximum HR (18 features). The difference in mean and CV SpO 2 (SpO 2 mean diff and SpO 2 CV diff), and difference in mean and CV HR (HR mean diff and HR CV diff), between Day A-1 and Day A-2 were also examined (4 features). Pearson's correlation coefficient was calculated for each possible pair of features.
During the eight-day period (days 3-10), 70 patients experienced SpO 2 alerts across 149 monitoring days. To develop a model to predict SpO 2 alerts days, validated using cross-validation, data for these 149 alert days were used with data for 157 monitoring days with no SpO2 alerts, Fig. 1. The 70 participants who experienced an alert during the monitoring period recorded data on 354 days where an SpO 2 alert did not occur. Ninety-seven of these 354 days were identified with no SpO 2 alerts on the two preceding or two subsequent days (from 34 participants). Data for 60 monitoring days (from 49 patients) were also randomly selected from the 126 patients (865 days) who did not experience any alerts, Fig. 1. The final model development dataset included 306 patient days, with data from 119 patients.
Tenfold cross-validation was used to develop the predictive model, and to assess model performance, with feature selection and model optimisation performed within each fold [26,27]. Data were stratified into ten folds, ensuring that data for each individual participant were present in one fold only (9 folds included data for 12 participants, and 1 fold included data for 11 participants). The number of patients days in each fold ranged from 25 to 37 days, with a mean (standard deviation) of 30.6 (3.9) days.
Sequential forward feature selection was implemented within 10fold cross-validation to select the optimum combination of five features to predict days with SpO 2 alerts. Bayesian optimisation was used to tune the number of learning cycles for each random forest classifier. These measures were implemented to increase the interpretability and robustness of the final model. A final model was then developed by resubstituting all data.
Model performance was assessed using the mean area under the receiver operating characteristic (AUC ROC), sensitivity and specificity across all folds. Sensitivity, or the true positive rate, was calculated as the number of correctly predicted alerts divided by the total number of alerts. Specificity, or one minus the false positive rate, was calculated as the number of correctly predicted non-alert days, divided by the total number of non-alert days. The specificity of the predictive model was additionally examined using the remaining data for 805 patient days recorded by participants who had no alerts during the monitoring period (Fig. 1). Data analysis and model development were conducting using custom-developed scripts in Matlab (Mathworks, Natick, MA, USA).

Data summary
Data for up to ten days following discharge were examined, resulting in 1368 patient days. On the day of hospital discharge, 209 patients used the remote monitoring system, with this number reducing over the tenday period to 134 active users on day 10 (Fig. 2). Over the ten-day monitoring period, the total dataset included 1378 days with breathlessness scores entered by 190 patients, 1775 days with SpO 2 measurements by 209 patients, and 1763 days with HR measurements by 206 patients. 196 patients used the remote monitoring system for at least three days (Table 1), with SpO 2 and HR data for these patients used to develop a model to predict SpO 2 alerts.
Age and sex were optional entries in the monitoring system, 114 patients reported both their age and sex (39.5 ± 13.9 years; 74 female), Table 1. The effects of age and sex on SpO 2 alerts were examined using data for these 114 patients.

Data analysis
Descriptive statistics of the daily mean and minimum SpO 2 , and mean and maximum heart rate and breathlessness, for the ten days after hospital discharge are presented in Table 2.
Mean and minimum SpO 2 , and mean and maximum heart rate, increased significantly over the ten days following hospital discharge (p < 0.005 for all), while the mean and maximum breathlessness decreased over the ten days (p < 0.005 for both), Fig. 3. The CV of SpO 2 , HR and breathlessness decreased significantly over the ten days, Fig. 3.
The effect of patient group (Sym, Asym, NoAlert), age and sex on overall mean SpO 2 and overall mean heart rate are presented in Table 3, Fig. 4a and 4b. The effect of patient group on age and sex are also presented in Table 3, Fig. 4c and 4d.

Predictive model
Sequential forward feature selection selected five features for inclusion in the final model (from highest to lowest feature importance): SpO 2 min Day A-2 , SpO 2 min Day A-1 , SpO 2 CV diff, HR CV Day A-2 , HR CV Day A-1 .
The random forest classifier predicted an SpO 2 alert on a given day with a mean sensitivity of 66.00 ± 18.57 % and mean specificity of 88.31 ± 10.97 %, and mean AUC ROC of 0.80 ± 0.11 across tenfold cross-validation, Fig. 5. The mean specificity of the predictive model on all unseen non-alert days was 88.72 ± 1.99 %.
Seventy-two out of 101 symptomatic alert days were correctly predicted as alert days, 32 out of 48 asymptomatic alert days were correctly Fig. 2. The number of patients, N, who entered data for SpO 2 , heart rate, and breathlessness, and who experienced an SpO 2 alert (SpO 2 < 94 %), in the 10 days following hospital discharge.

Table 1
Summary of recorded SpO 2 alerts in the full cohort, and in the subset of the cohort where age and sex data are available. * This group consists of all the patients who experienced only symptomatic alerts, together with a subset of the patients who experienced asymptomatic alerts. ** This group consists of a subset of the patients who experienced asymptomatic alerts.   Fig. 3. Data for all patients for the first ten days after discharge from hospital. Mean, coefficient of variation (CV) of all SpO 2 , heart rate and breathlessness data, and minimum SpO 2 , maximum heart rate and maximum breathlessness, for each day across all patients are presented. The mean and standard error of each feature across all patients are presented.

Table 3
The effect of patient group (Sym/Asym/NoAlert) on mean SpO 2 , mean HR, age and sex. Mean SpO 2 and mean heart rate are calculated for each patient using all data for that patient during the monitoring period. Significant differences (p < 0.015) are indicated using bold font. predicted as alert days, and 147 out of 157 non-alert days were classified correctly. Correlation analysis revealed that several SpO 2 and HR features were strongly correlated with each other, with Rho > 0.8 or Rho < -0.8, Table 4.

Discussion
A model to predict SpO 2 alerts (<94 %) in patients with COVID-19 in the ten days following hospital discharge was developed using random forest classification. The model utilised patient-recorded SpO 2 and HR features measured on the two preceding days, and the day of discharge. The random forest model provides a method to predict low pulse oxygen saturation a day in advance of the SpO 2 event occurring. Model performance was assessed using cross validation, with a mean ROC AUC of 0.80, true positive rate of 66 % and a low false positive rate of 11.69 %. The presented method could allow early intervention to prevent further clinical deterioration and assist clinical teams to remotely triage patients following hospital discharge.
Prediction of low SpO 2 in patients with COVID-19 using remote monitoring data after hospital discharge has not previously been reported. However, several studies have presented models to predict clinical deterioration in hospitalised patients with COVID-19 [13][14][15][16][17][18][19][20], or in non-hospitalised individuals with COVID-19 [21,22]. A range of machine learning techniques models using clinical and laboratory data (including SpO 2 ) obtained on hospital admission have been reported, including logistic regression [14,16,17], deep learning [13], random forests [16,20] or gradient boosting models such as XGBoost [16,18,20] or CatBoost [19,20]. The random forest algorithm was used in this study due to its interpretability, and the performance of previously reported models based on ensembles of decision trees to predict clinical decline in hospitalised patients with COVID-19 [18][19][20] and to detect COVID-19 infection in non-hospitalised individuals with COVID-19 [21]. The random forest model presented here, with a ROC AUC of 0.80, performed comparably to previous studies predicting clinical deterioration in patients or non-hospitalised individuals with COVID-19 [13][14][15][16][18][19][20][21][22][23]. To demonstrate the generalisability of predicting low SpO 2 in COVID-19 patients following hospital discharge using the methods presented, three alternative machine learning algorithms were also implemented (Supplementary Material): logistic regression with LASSO regularisation, support vector machine, and adaptive boosting. These alternative models achieved cross-validated ROC AUCs of 0.75 ± 0.11 for logistic regression, 0.79 ± 0.06 for adaptive boosting and 0.83 ± 0.07 using support vector machines (Supplementary Material), indicating that the performance of the presented method would generalise to other classification algorithms.
Remote monitoring also provides an opportunity to examine how HR, SpO 2 and breathlessness vary in the period following hospital discharge. Mean HR significantly increased over the ten day period examined. Consistent with this, a previous study reported that heart rate in individuals with COVID-19 decreased until approximately 13 days after symptom onset, and increased after this point [22]. Mean SpO 2 also significantly increased, while breathlessness significantly decreased, consistent with symptom improvement following discharge for the cohort in general. The CV of daily SpO 2 , HR and breathlessness measures decreased over the ten days, indicating increased stability in these measures over time.
No significant effects of age or sex on the occurrence of  asymptomatic hypoxia in COVID-19 were observed. A previous study of 195 patients with COVID-19 at a hospital emergency department reported silent hypoxia in 25 (13 %) of patients [25]. In the current study, 39 patients (19 %) experienced asymptomatic SpO 2 alerts at least once during the ten days after discharge. The slightly higher preponderance of asymptomatic alerts is potentially due to differences in clinical setting, inpatient vs post discharge, and resulting differences in SpO 2 thresholds, 94 % for early identification of clinical decline after hospital discharge in this study, consistent with previous studies monitoring patients after hospital discharge [5,7], while a lower threshold of 90 % was used in the previous hospital-based study [25]. The inclusion of elevated HR (>100 bpm) as a symptom of respiratory distress in this study, consistent with previous studies on post-discharge monitoring in COVID-19 [5], may also have contributed to this difference. The generalisability of the presented predictive model is limited by the cohort size, particularly in the subset of the cohort who provided demographic information. However, the cross-validated classification results indicate potential as a clinical tool which warrants trialling in a larger cohort. It should be noted that several features were highly correlated, Table 4, indicating that alternative feature combinations may produce similar performance to those reported. There was a possibility for data entry errors for breathlessness scores, however, the Bluetooth connection of the pulse oximeter reduced the input errors in SpO 2 and HR measures. While the accuracy of SpO 2 compared to blood oxygen saturation in COVID-19 has been reported to be adequate, with a reported bias of 0.4 % [32], physiologic or mechanical artifacts with vital sign alerts have also been reported, together with methods to identify erroneous alerts [33].
This study demonstrates the feasibility and effectiveness of predicting SpO 2 < 94 % in recently discharged COVID-19 patients on a given day, based on data for the two previous days and the day of hospital discharge. The presented model may provide a useful tool to enhance remote monitoring systems, providing early identification of patients at risk of low pulse oxygen saturation.

Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Statement on conflicts of interest.
Colin Edwards is Chief Scientific Officer for patientMpower Ltd., who developed the remote monitoring system used in this study.

Summary table
What was already known on this topic?
• There is a risk of clinical deterioration after hospital discharge in patients with COVID-19, and remote monitoring of vital signs may provide a method to improve patient outcomes during this period. Pulse oxygen saturation (SpO 2 ) below 94 % indicates hypoxia and clinical deterioration in patients with COVID-19.
Not all COVID-19 patients experience breathlessness, or elevated heart rate during low SpO 2 events.
A model to predict low SpO 2 events following hospital discharge using a remote monitoring system has not previously been presented. What has this study added to our knowledge? • Changes in SpO 2 , heart rate and breathlessness measures after hospital discharge in COVID-19 have been reported for the first time.
Age and sex did not affect symptom presentation during low SpO 2 (<94 %) events.
A new model to predict low SpO 2 (<94 %) in COVID-19 patients after hospital discharge, based on remote monitoring data for the two previous days is reported.

Table 4
SpO 2 and heart rate features with strong positive (Rho > 0.8) or negative (Rho < -0.8) correlations. Features included in the final random forest model are in bold font.