{'fraction_non_alphanumeric': 0.032326395284582796, 'fraction_numerical': 0.016117148646159513, 'mean_word_length': 4.872904272579773, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Forecasting Emergency Department Capacity Constraints for COVID Isolation Beds

Predicting patient volumes in a hospital setting is a well-studied application of time series forecasting. Existing tools usually make forecasts at the daily or weekly level to assist in planning for staffing requirements. Prompted by new COVID-related capacity constraints placed on our pediatric hospital's emergency department, we developed an hourly forecasting tool to make predictions over a 24 hour window. These forecasts would give our hospital sufficient time to be able to martial resources towards expanding capacity and augmenting staff (e.g. transforming wards or bringing in physicians on call). Using Gaussian Process Regressions (GPRs), we obtain strong performance for both point predictions (average R-squared: 82%) as well as classification accuracy when predicting the ordinal tiers of our hospital's capacity (average precision/recall: 82%/74%). Compared to traditional regression approaches, GPRs not only obtain consistently higher performance, but are also robust to the dataset shifts that have occurred throughout 2020. Hospital stakeholders are encouraged by the strength of our results, and we are currently working on moving our tool to a real-time setting with the goal of augmenting the capabilities of our healthcare workers.


Introduction
The emergency department's (ED) census, the total number of patients admitted or waiting to be admitted, contains recurring patterns such as seasonality and hourly trends (Flottemesch et al., 2007;Nas and Koyuncu, 2019). Though classical time series approaches have been shown to work relatively well for predicting patient volumes (Tandberg and Qualls, 1994), there is still scope for improvement. Most existing census forecasts tend to use daily or weekly time frames, as these allow for staffing decisions to be made in advance ( Though the existing literature is large, our research provides the following contributions. First, we show that parametric models are subject to a performance collapse when COVID causes a dataset shift to our time series. Second, a small training set and an implicit transfer learning paradigm is demonstrated to obtain the best performance. Third, GPRs obtain high prediction accuracy at the hourly frequency for our hospital's pediatric census. GPRs are a natural fit for our modelling task as they 1) can be easily calibrated through their probabilistic inference, and 2) allow for kernels to be constructed that match our census' statistical patterns. Lastly, we show how a machine learning (ML) tool can be developed and evaluated to address a clinical use case in the COVID-era.

Data
All ED visits from June 1st, 2018 to August 30th, 2020 were extracted from our hospital's Epic database, amounting to 159,683 unique patient visits. Patient arrival and discharge time were used to calculate the maximum census of ED patients in any one hour. Additional demographic, physiological, and hospital information was processed. Categorical features, like sex, were calculating as an hourly share, and continuous features, like blood pressure, as an hourly average. Appendix A provides a full description of all preprocessing steps.
Since the onset of COVID, our pediatric hospital's ED has operated under strict capacity limits with a total of 47 isolation beds. Any patient showing a possible COVID symptom must be placed in one of these areas. Since common symptoms like a runny nose or fever are sufficient for a patient to be placed in an isolation room, a large fraction of the census use up these scarce resources.
In our analysis we assumed that 100% of patients required an isolation bed. Though this will not be the case in practice, we lacked historical data to determine this share in any one hour and therefore opted to be conservative in the estimation of our ED's capacity constraints. Figure 1 shows the trend in the maximum number of hourly patients in our ED since March 2020. As the ED census moves from the normal state (≤ 30) into escalation level 1 (31-37), level 2 (38-47), or level 3 (≥ 48), a variety of prescribed actions are carried out to free up additional hospital space and martial more resources. Under our 100% isolation assumption, if the escalation level is 3, then our ED will be unable to admit any new patients. This represents a significant harm to our patients, community, and healthcare workers. Our hospital will benefit from an early warning system of impending escalation level changes.

Modelling and evaluation
Starting on January 1st, 2020 models were required to make a rolling forecast every hour, for a 24 hour window, with retraining occurring at midnight. For example, the training set available to a model for February 3rd, 2020 would include all hourly data from June 1st, 2018 until midnight on February 2nd, 2020. The GPR was benchmarked against a Lasso model provided with autoregressive (AR) features (Rasmussen and Williams, 2005;Tibshirani, 1996), as well as one weighted by local observations (see Appendix B for a full list of features). The AR-Lasso used features that were of the current hour, as well as those of the previous ten hours. The level of regularization was selected to bound the number of false positives to 10% (Drysdale et al., 2019). For the locally weighted AR-Lasso, a validation set was created by taking the week of data before the forecasting day, and selecting a length scale for the radial basis function kernel which minimized that week's mean square error (MSE).
The GPR was fit with the GPyTorch package (Gardner et al., 2018) using an Adam optimizer (learning rate = 0.1), and was trained until the change in marginal log-likelihood was less that 1e-3. At the time of nightly retraining, the previous day's weights were loaded (the hyperparameters of the various kernels). The GPR kernel combined multiple kernel types (linear, cosine, and RBF) to capture cyclical trends, up-weight recent observations, and match hours that had similar features (see Appendix C for a complete specification of the kernel). The feature set used by the GPR comprised of the time series for previous values of the regression label (the maximum number of patients in a given hour), the number of arrivals and discharges, the hour of the day, the composition of patients by acuity score and arrival method, as well as the number of physicians in the ED. These features were chosen as they were amenable for real-time extraction and consistent with clinical expertise. For each regression method, a separate model was trained for a given forecasting lead.
All one-day-ahead forecasting results were analyzed from March 1st to August 31st. Point estimates were evaluated by calculating the daily R-squared (R2) for each forecasting lead. Due to the importance of our hospital's capacity constraints and prescribed action required for different escalation levels (see Figure 1), we evaluated the performance of our model by how well it predicted positive changes to the current escalation level. For example, if the ED census was 32 at the current hour (level 1), and the 6-hour forecasting lead predicted 49 patients (level 3), this would be a predicted positive change of two levels. Predicted positive changes are therefore forecasts where the level of patients corresponds to a higher escalation level than what is seen at the current hour. By treating the differences in the levels as different classes, precision and recall could be calculated. Specifically, there is a class for positive level changes of one, two, or three, relative to the current level. A less stringent class is that of any positive level change (i.e. one, two, or three).

Results
The AR models showed a fragile one-day-ahead forecasting performance. Figure 2(a) demonstrates this phenomenon for the 4-hour lead (other leads showed the same dynamic). As patient volumes fell in March, the AR-Lasso actually saw an increase in its root MSE (RMSE), despite the smaller variation in the label, for two reasons. First, the expected level of patients was systematically too high, and second, the relative ranking deteriorated as measured by the concordance index. Even after the RMSE of the AR-Lasso began to cover, its performance never converged to the level of the GPR. Although the locally weighted model performed better than the AR-Lasso, it still lagged behind the GPR.
Due to the non-stationary nature of the ED census, the optimal size of training data for the GPR was also considered. We found that using 1-2 months of training data yielded a much lower performance (≈ 65% R2) compared to 1-2 weeks (≈ 80% R2) or even 1-3 days (≈ 85% R2). Excluding the 1-hour lead, the performance for leads 2-24 was similar, on average. Even though a short time horizon is clearly beneficial for adapting to dataset shifts, the magnitude of the performance obtainable using only a few training days was surprising. However an examination of the relative performance differences from iterative retraining (using the previous day's hyperparameters) compared to fitting the GPR from scratch revealed that the latter, de novo model training, would result in a significant performance decline (≈ 7.5% R2) as Figure 2 Using point estimates of the GPR, the model was able to accurately predict a positive change in the escalation level (e.g. going from a normal state (≤ 30 patients), to level 2 (38 − 47 patients)) as measured by both precision and recall. Figure 3(a) shows that the average precision was between 50-75% for predicting the exact (positive) change in escalation level, and less stringently, 80% for any positive change. Because the GPR provides a probabilistic output, a trade-off between precision and recall can be obtained by using different percentiles of the output. Figure 3(b) shows that sensitivity can be traded off between 20-70% depending on a tolerance for false positives for most forecasting leads. Conditional on a given recall, precision precision shows a U-shaped relationship for the forecasting lead. This is due to label balance peaking at the 12 hour lead, as figure 3(c) shows.

Conclusion
The GPR provides sufficient flexibility to handle non-stationary ED census data in the COVID-era and obtains a strong performance, especially when compared to more brittle parametric models. Using a small number of training days, having nightly retraining, and loading pre-existing weights obtains the best performance. We have shown how model performance can be explicitly linked to a clinical application in our hospital, namely: predicting escalation levels to allow for early operational decisions. Our hospital stakeholders are impressed with the results, and are eager to test our tool in a real-time setting. These forecasts show clear potential to augment and enhance the capabilities of healthcare workers.