{'fraction_non_alphanumeric': 0.03570006635700066, 'fraction_numerical': 0.012408759124087591, 'mean_word_length': 5.179171791717917, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Does Deep Learning REALLY Outperform Non-deep Machine Learning for Clinical Prediction on Physiological Time Series?

Machine learning has been widely used in healthcare applications to approximate complex models, for clinical diagnosis, prognosis, and treatment. As deep learning has the outstanding ability to extract information from time series, its true capabilities on sparse, irregularly sampled, multivariate, and imbalanced physiological data are not yet fully explored. In this paper, we systematically examine the performance of machine learning models for the clinical prediction task based on the EHR, especially physiological time series. We choose Physionet 2019 challenge public dataset to predict Sepsis outcomes in ICU units. Ten baseline machine learning models are compared, including 3 deep learning methods and 7 non-deep learning methods, commonly used in the clinical prediction domain. Nine evaluation metrics with specific clinical implications are used to assess the performance of models. Besides, we sub-sample training dataset sizes and use learning curve fit to investigate the impact of the training dataset size on the performance of the machine learning models. We also propose the general pre-processing method for the physiology time-series data and use Dice Loss to deal with the dataset imbalanced problem. The results show that deep learning indeed outperforms non-deep learning, but with certain conditions: firstly, evaluating with some particular evaluation metrics (AUROC, AUPRC, Sensitivity, and FNR), but not others; secondly, the training dataset size is large enough (with an estimation of a magnitude of thousands).


I. INTRODUCTION
With the widespread adoption of Electronic Health Records (EHR), there is an increased emphasis on predictive models that can help clinical decisions and treatment recommendation. EHR contains various types of data, including demographics information, laboratory values, vital signs, physiology signals and treatment records, which are generally sparse, irregularly sampled, and multivariate. With the development of machine learning methods, especially deep learning and neural networks, more research focuses on clinical prediction or physiology management estimation with EHR using machine learning methods [3]. Deep learning shows excellent capabilities to extract information and features from big data, but the performance of deep learning depends on the training dataset size, model structure, data quality, and similar other related conditions. Many research or actual work fails to reach the ideal target when utilizing deep learning models [4]. In some cases, non-deep learning, such as XGBoost has a similar or even better performance with deep neural networks [5].
In this paper, we systematically analyze the machine learning models' performance on the clinical prediction domain, using EHR, especially the physiology time series data. We propose a general framework to handle the raw multivariate time series data. We utilize the Physionet 2019 challenge public dataset [1] to predict Sepsis outcomes in ICU units, defined as a binary classification problem. And we train and evaluate the datasets with 10 baseline machine learning models: 3 deep learning models: (1) InceptionTime [6], (2) TCN (Temporal CNN) [7] and (3) Transformer [8] and 7 non-deep learning models: (4) LR (Logistic Regression), (5) SVM (Support Vector Machines), (6) KNN (K Nearest Neighbors), (7) GP (Gaussian Process), (8) DT (Decision Tree), (9) RF (Random Forests) and (10) XGBoost [5]. Then, we evaluate the performance of different machine learning models using 9 different evaluation metrics with specialized clinical meanings. Also, as the training dataset size is a crucial aspect for the models' performance, we sub-sample the training dataset and present a quantitative analysis of the relationship between the training dataset size and the performance of machine learning models using the learning curve fit method.
Based on the proposed systematic and quantitative analysis framework, we find that deep learning indeed outperforms non-deep learning only under certain conditions.
• The four evaluation metrics: AUROC, AUPRC, Sensitivity, and FNR show that deep learning outperforms non-deep learning. For example, AUROC is greater than 0.9 for deep learning models, while it is less than 0.8 for non-deep learning models. • Labeled training dataset size: The larger training dataset improves the performance of the machine learning models, and deep learning models need a certain number of training dataset sizes to reach superior performance. The threshold value depends on different evaluation metrics. The overall estimate of training dataset sizes is about a magnitude of thousands.

II. RELATED WORK
There is an extensive body of clinical prediction based on EHR data, especially the physiology time series. Hatib et al. [9] use feature engineering to extract features from arterial pressure waveform to the prediction of an upcoming hypotensive event. Lee et al. [10] extract basic statistical features and then uses feedforward neural networks to predict mortality.
Some recent works try to extract the information or features from raw time series data, without feature engineering or statistical analysis. Kok et al. [7] use TCN for automated prediction of Sepsis outcome. Che et al. [2] develop LSTM for Sepsis prediction with missing values in raw data. Fawaz et al. [6] provide InceptionTime (a CNN-based model) to deal with time series classification problem. Zhao et al. [4] use the InceptionTime model to analyze the intraoperative time-series monitoring data, predicting the post-hysterectomy quality of recovery. Wu et al. [8] focus on the Transformer model and use it in the influenza prevalence prediction case.

III. METHODS
A. Data analysis and preprocessing 1) Data characteristics: This work is motivated by the analysis of physiological time series data in EHR, which are generally sparse, irregularly sampled, and multivariate. The EHR data contains the real-time physiological status of patients [3], such as vital signs or other physiology signals, laboratory values, demographics information, treatments, and outcomes. The data contains continuous and categorical data with varied time lengths and different sampling frequencies.
The physiology time series are usually sparse, with missing data both in the time dimension and feature dimension. Also, the positive sample size is generally much smaller than the negative one in most clinical prediction domains. The imbalanced dataset is a fundamental problem in the clinical prediction field.
2) Public datasets: Physionet 2019 challenge dataset (early prediction of Sepsis from clinical data) includes hourly physiological time series data consisting of 8 vital signs, 26 laboratory values, and 6 demographic details of 40,336 samples from the MIMIC III database [1]. The time length varies from 8 to 336 time-steps. The outcome is the onset of sepsis.
3) Preprocessing: a) Missing data: The physiology data is inconsistent in the timeline. Meanwhile, missing values are correlated with feature labels. As the physiology data reflect the clinical status of patients, we assume that the values vary little within a small time window. Thus, we use forward and backward fill for the missing points in the time domain. When the missing value relates to the feature labels, we fill other specific values with a special meaning, e.g., -1 or N/A. b) Various data length: Data length depends on time length and sampling frequencies. The time lengths of physiology time series are significantly different among features or patients. Also, the sampling frequencies are different among physiology features. The data filling process includes two steps: 1) firstly, utilize forward and backward filling according to the max sampling frequency. 2) fill each sample according to the max length of all samples with a special meaning, e.g., -1 or N/A. c) Continuous and categorical data: Generally, vital signs and laboratory values are continuous data and we use normalization to maintain the contribution of each feature and keep the model unbiased. Demographics information is mainly categorical data and we use one-hot encoding to convert the categorical data. After the filling, normalization, and one-hot encoding, processed EHR data is formatted for training and testing.
d) Imbalanced dataset: The imbalance issue is a common problem in the clinical domain: positive samples are much less than negative ones. For example, the Sepsis predction dataset has 40,336 samples with 2,932 positive patients, with a rate of 7.27%. Two main methods deal with the imbalance dataset: data augmentation and special loss functions. As the data augmentation methods affect the dataset size, we use Dice Loss [2] as the loss function of the prediction problem.
B. Models 1) Deep learning models: Deep learning models are based on neural networks that are composed of a large number of layers and parameters, which can be potentially used for extracting features from time series and downstream tasks such as regression, classification, forecasting, and feature embedding. We employ 3 different deep learning models to directly tackle the physiology time series for Sepsis prediction task, including InceptionTime [6], TCN (Temporal CNN) [7] and Transformer [8].
a) InceptionTime: InceptionTime is a 1D-CNN-based deep neural network. It consists of a series of Inception modules followed by a Global Average Pooling layer and a Dense layer with a softmax activation function. In each Inception module, multi-dimension time series data are transformed into 1D data. Three one-dimension filters with lengths of 10, 20, and 40, respectively, are applied simultaneously to the output. Residual blocks are added to mitigate the vanishing gradient problem. b) TCN: TCN is a variant of the CNN and employs ordinary convolutions and dilations. It is suitable for sequential data with temporality and large receptive fields. The latent correlation among series can be learned from this model. c) Transformer: The Transformer-based time series model consists of Transformer encoder layers.The network is composed of an input layer, a positional encoding layer, and a stack of transformer encoder layers. We use one transformer encoder layer in this work.
2) Non-deep learning models: For Non-deep learning models, we evaluate 7 models with the statistical information extracted from raw time series data. Some machine learning models are inconvenient to be used with raw time-series data, such as logistic regression. Usually, the preprocessing is needed to extract the basic (descriptive) statistical information or feature from the time-series data to be used in the machine learning methods [9], [10]. The statistical information includes the maximum, minimum, average, median, standard deviation, quantiles, moments, etc.  100, 200, 400, 600, 800, 1000, 2000, 4000, 6000, 8000,  10000, 15000, 20000, 25000, 30000, 36303.
2) Performance metrics and evaluation: For the classification task, the most commonly reported metric is the area under the receiver operator characteristic curve (AU-ROC), which combines the information from sensitivity and specificity into a single value. We also report AUPRC since it is a good metric for describing model performance given imbalanced datasets. And we evaluate the accuracy, sensitivity (recall, TPR (true positive rate)), specificity (TNR, 1-FPR (false positive rate)), FNR, FPR, NPV, PPV, which represent actual clinical significance [11].
3) Training dataset size impact: The evaluation results of machine learning models are largely dependent on the training dataset size. We choose the learning curve approach [12] to explore the relationship between the performance and the training dataset size. The curve model is represented as an inverse power law function given by Using the evaluation results of the models trained with different training datasets sizes, we calculate the unknown parameters (b1 and b2) with nonlinear weighted least squares optimization using the following equation.
where r p is the residual between the real value and the fitted function.

IV. RESULT AND DISCUSSION
A. Comparison of machine learning models' performance Our findings include: • The deep learning models outperform the non-deep learning methods for the AUROC, AUPRC, sensitivity, and FNR.
• For the accuracy, specificity, NPV, and FPR, both deep learning models and non-deep learning models have similar performance results. • The performance of PPV evaluation metric differs from diverse machine learning models. • All machine learning models could be divided into three groups based on performance: (1) deep learning models; (2) XGBoost, Random Forest, and Decision Tree; (3) Gaussian process, KNN, SVM, and logistic regression. The performance is ranked in the order of best to worst: Group 1, Group 2, Group 3. • Among non-deep learning models, XGBoost performs best compared to the other non-deep learning models. XGBoost could not perform as well as deep learning methods though.

B. Training dataset size and learning curves
We also analyze the relationship between the training dataset size and the model performance. Fig. 2 depicts the impact of the training dataset size on the performance of the models, for the AUROC (upper left), AUPRC (upper right), Sensitivity (below left), and FNR (below right). Each line represents a model's performance according to the increasing number of training dataset sizes.
Our findings include: • When the training dataset size is small, the deep learning models do not outperform the non-deep learning models, especially for the sensitivity evaluation metrics. • With the increased training dataset size, the deep learning models show an excellent ability comparing with non-deep learning models for all four evaluation metrics. • The increased number of training dataset size has a larger impact on Group 1 than Group 2, while no influence on Group 3. • With the increased training dataset size, all the models' performance approach saturation.

V. CONCLUSIONS
Our systematical and quantitative study showed that deep learning indeed outperforms non-deep learning with specific training dataset size (an estimation of a magnitude of thousands) and some particular evaluation metrics (AUROC, AUPRC, Sensitivity, and FNR). No general effect of machine learning models or deep learning models was observed with small sample sizes (below estimation of a magnitude of thousands). Therefore, deep learning has the potential to be used as a tool for clinical prediction.
The future direction of our work will extend our quantitative analysis to investigate the relationship between the clinical research performance and the scale of the network, data points, the sampling frequency of raw data, augmentation of data, etc. It can assist researchers to determine the models and the dataset quality for clinical prediction research based on machine learning approaches.