{'fraction_non_alphanumeric': 0.027932672449401074, 'fraction_numerical': 0.011204047914085089, 'mean_word_length': 5.135254988913526, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Novel Method of Heart Failure Prediction Based on DPCNN-XGBOOST Model

The occurrence of perioperative heart failure will affect the quality of medical services and threaten the safety of patients. Existing methods depend on the judgment of doctors, the results are affected by many factors such as doctors’ knowledge and experience. The accuracy is difficult to guarantee and has a serious lag. In this paper, a mixture prediction model is proposed for perioperative adverse events of heart failure, which combined with the advantages of the Deep Pyramid Convolutional Neural Networks (DPCNN) and Extreme Gradient Boosting (XGBOOST). The DPCNN was used to automatically extract features from patient’s diagnostic texts, and the text features were integrated with the preoperative examination and intraoperative monitoring values of patients, then the XGBOOST algorithm was used to construct the prediction model of heart failure. An experimental comparison was conducted on the model based on the data of patients with heart failure in southwest hospital from 2014 to 2018. The results showed that the DPCNN-XGBOOST model improved the predictive sensitivity of the model by 3% and 31% compared with the text-based DPCNN Model and the numeric-based XGBOOST Model.


Introduction
Heart failure has been considered as one of the deadliest human diseases worldwide, and the accurate prediction of this risk would be vital for heart failure prevention and treatment. However, the simple early warning system of adverse events often cannot catch signs of heart failure adverse events in time. Once the adverse events occur, the disease is serious or terminal, resulting in the difficult treatment and the limited effect of intervention. It is of great scientific significance and social value to actively develop the risk prediction of heart failure adverse events, which is helpful in the early warning and intervention clues of adverse events. Recently, artificial intelligence technology has been widely used in the medical field [Hannun, Rajpurkar, Haghpanahi et al. (2019)  ]. Currently, the research on heart failure is mainly based on the data from patients' medical records, physical characteristics, auxiliary examination, the treatment plan, and the algorithm is used to build the model for studying, analyzing and classifying of diagnosis and prediction. In addition, most studies mainly analyzed the characteristics of electrocardiogram data and built the diagnostic model of heart failure [Yu and Lee (2012); Masetic and Subasi (2016); Melillo, Izzo, Orrico et al. (2015); Acharya, Fujita, Sudarshan et al. (2017)]. Zheng et al. [Zheng, Zhang, Yoon et al. (2015)] presented a method used support vector machine algorithm to analyze the data of patients with heart failure, including age, type of medical insurance, sensitivity assessment (audio-visual and thinking), complications, emergency treatment, the druginduced risks, the period of last hospitalization, and built a prediction model for the readmission of patients with heart failure, with a prediction accuracy of 78.4%. Choi et al. [Choi, Schuetz, Stewart et al. (2017)] used the recurrent neural network algorithm to analyze the diagnostic data of patients with heart failure, including time series of doctor's orders, spatial density and other characteristics, to build a diagnostic model of heart failure, and verified by experiment that the area under the curve (AUC) of the diagnosis of this model was 0.883. Chen et al. [Chen, Zheng, Li et al. (2016)] analyzed 24 hours dynamic electrocardiogram of heart failure patients and healthy controls by using support vector machine (SVM) algorithm based on non-equilibrium decision tree. Shameer et al. [Shameer, Johnson, Yahi et al. (2017)] also utilized Naive Bayes algorithm to analyze about data variables of patients with heart failure, including diagnosis data, treatment data, examination data, records of doctor's orders, and vital signs data, and built a model for predicting readmission of patients with heart failure, with a predicted AUC of 0.78. However, these studies only focused on the numerical data of preoperative test or intraoperative monitoring. Its prediction of poor timeliness, low accuracy, and non-fusion of patient heterogeneity data. Specifically, the heterogeneity of patients are embodied in many aspects, such as the age, occupation, gender, various physiological indices [Koerkamp, Stijnen, Weinstein et al. (2011)] as well as various types of numerical laboratory data, textual diagnostic information. These indices are used to predict heart failure, the results are more practical and targeted. Therefore, in order to make the results of research and analysis more accurate and more convenient to apply to practice, in this paper, we use Deep Pyramid Convolutional Neural Networks (DPCNN) and Extreme Gradient Boosting (XGBOOST) method to model the risk prediction of heart failure after operation based on the preoperative and Intraoperative medical data of patients, so as to construct a scalable, low-cost and effective prediction solution for heart failure. Our main contributions are in two areas: 1) A novel heart failure prediction framework is proposed, which integrates the advantages of deep neural network and XGBOOST model, and uses the structure and unstructured data of patients before and during operation to predict heart failure. 2) Experiments based on real heart failure data show that fusion of multi-source heterogeneous data can improve the accuracy of heart failure prediction. The rest of this paper is organized as follows: The preliminary and related technology, and methodology of this paper is discussed in Section 2. Section 3 reports the experimental results and discusses the implications of the study. Finally, Section 4 discusses the conclusion of this paper.

Deep Pyramid Convolutional Neural Networks (DPCNN)
Deep Pyramid Convolutional Neural Networks (DPCNN) [Johnson and Zhang (2017)] is a low-complexity word-level deep convolutional neural network architecture for text categorization [Zhang and Wallace (2017), Liu, Qiu and Huang (2016)] that can efficiently represent long range associations in text. The specific structure is shown in the Fig. 1. The first layer performs text region embedding, which generalizes commonly used word embedding to the embedding of text regions covering one or more words. It is followed by stacking of convolution blocks (two convolution layers and a shortcut) interleaved with pooling layers with stride 2 for down sampling. The final pooling layer aggregates internal data for each document into one vector. It uses max pooling for all pooling layers. The DPCNN can effectively extract the features of long-distance relationship of the text with low complexity and better effect than the previous CNN structure. Therefore, we use this model to extract the features of unstructured diagnosis text of patients to predict heart failure. XGBOOST [Chen and Guestrin (2016)] is used to extract the numerical features of patients, which are an integrated learning method proposed by Tianqi Chen based on GBDT [Ye, Chow, Chen et al. (2009)]. The improvement of XGBOOST algorithm to GBDT algorithm lies in that the second derivative is used to calculate the objective function in the process of model optimization, besides, the regularization term is added to the objective function to prevent the algorithm from over-fitting in the training process, moreover, XGBOOST algorithm uses the idea of random forest for reference in the training process, and does not use all samples in the iteration process, and does not use every iteration. The generalization ability of the model is effectively improved by sampling all the features of the samples and training some of the features of the samples. Different patients will have different preoperative tests and intraoperative monitoring according to different diseases. In order to facilitate the construction of data set, this paper integrates the test indicators and intraoperative monitoring indicators of all patients, so a large number of null values will be generated in the data set. XGBOOST can treat the missing values as a sparse matrix, so this paper uses XGBOOST model for prediction modeling.

DPCNN-XGBOOST models
The DPCNN-XGBOOST heart failure prediction model proposed in this paper is shown in Fig. 2, which combined with the advantages of DPCNN and XGBOOST. The DPCNN was used to automatically extract features of patients' diagnostic texts, and the text features were integrated with the preoperative examination and intraoperative monitoring values of patients, then the XGBOOST algorithm was used to construct the prediction model of heart failure. It includes three parts in Fig. 2.

Feature extraction of preoperative structured numerical data
Different patients will carry out different test items according to different diseases. In order to build standard data sets, this paper extracts indicators with common test attributes for positive patients and negative patients as feature items (see Tab. 1 for specific test attribute indicators). Due to the different dimension of test items, the test data of patients were standardized and normalized to facilitate data analysis, and the processed data can be classified and predicted as the input of XGBOOST. The top part of Fig. 2.   The feature extraction of preoperative unstructured text data is the text that extracting the previous medical illness, chief complaint-positive symptom/sign, history of present illness, echocardiogram, electrocardiogram, chest X-Ray, preoperative clinical diagnosis of patients (See Tab. 2 for specific patient examination text attributes). The text was preprocessed and converted into a word vector by using Tencent open-sources word vector [Song, Shi, Li et al. (2018)], then the predictive classification features of heart failure were extracted by DPCNN, finally forming a text vector of 250 dimensions. The middle part of the Fig. 2.

Feature extraction of intraoperative monitoring data
Intraoperative monitoring data are time series data, in this paper, we convert intraoperative monitoring data into the fixed features by the feature engineering methods. The methods used in the model include statistical outliers and statistical characteristics of patient monitoring attributes. The outliers are the time (in minutes) that the monitoring attribute data of patients exceed the normal interval, as shown in Tab. 3. The statistics include the maximum, minimum, mean, variance, standard deviation, kurtosis and skewness of each attribute. The formula is as follows. According to different types of operation and patients' diseases, the attributes of intraoperative monitoring are different. In order to construct the standard data set, this paper counts the intraoperative monitoring attributes of all patients (negative and positive), as the monitoring attribute column of patients during operation. When patients do not monitor an attribute, the value is empty, and finally the intraoperative monitoring data of patients is converted into a fixed dimension vector. (1)

Experimental and results
The experimental environment of this article was based on the server: Ubuntu 16.04 LTS was used as the operating system with Intel Xeon e5-2650 V4 processor and Nvidia GTX 1080 Ti GPU. The memory is 63 GB. Pytorch was used to build the Deep Pyramid convolutional neural network, and Python3.6 was used as the programming tool.

Database
The data used in this experiment were collected from surgery patient in hospital of China from 2014 to 2018. The number of patients in this data was 4700, including 536 positive patients, 4164 negative patients, 2718 female patients, and 1982 male patients. The age, gender and label distribution were shown in Figs. 3-5. The test set is randomly selected from the data set and accounts for 20% of the data set. In order to make the results statistically significant, the experimental method adopted 10-fold cross validation.

Performance evaluation
In this paper, the prediction of heart failure is modeled and analyzed as a dichotomy problem, and the performance of the model can be evaluated through the confusion matrix. According to the ground truth and prediction, true positive (TP), true negative (TN), false positive (FP) and false negative (FN) can be classified. Precision, recall and F1 Score were used to test the accuracy of classification.

Experimental and discussion
We report the experiments with DPCNNs and XGBOOST in comparison with previous models and alternatives. The specific experimental description is as follows. The code is publicly available on the internet.

Preoperative test and demographic data for heart failure prediction
Only preoperative test and demographic data were used to predict heart failure.   Due to the imbalance of data set (negative patient: 4164, positive patient: 536), the positive recall rate (sensitivity) of all models is low, but the weighted average accuracy (94%), weighted average recall rate (94%), F1 score (93%) and ROC value (92%) based on XGBOOST model are higher than other models. In addition, because there are a lot of missing values in the data set, the missing values are filled with 0 when using LR and SVM models, and the effect is worse than the tree model. Therefore, based on the preoperative patient test data, the tree model is more suitable for patients with missing value.

Prediction of heart failure with intraoperative monitoring data
Figure 8: XGBOOST ROC curve Figure 9: ROC curve of comparative model In this paper, the outliers time and statistical characteristics of monitoring data were extracted, and the model in Experiment 1 was used to predict heart failure. The experimental results are shown in Tab. 5. The ROC curve of the model is shown in Figs. 8 and 9.

Prediction of heart failure based on unstructured text data of Preoperative patients
Firstly, DPCNN is used to extract the text features of patients. Two convolution layers of blocks are used. Each convolution core is 3 × 3, the size of pooling layer is 2 × 2, and each layer has 250 convolution cores. Finally, a full connection layer is connected for heart failure classification and prediction. Tab. 6 of experimental results, ROC curve as shown in Fig. 10. It can be seen from the experimental results that the unstructured text data based on the preoperative examination of patients can better predict the postoperative heart failure of patients (sensitivity 90%, specificity 97%, ROC 98%). Analysis reason: the preoperative text includes the diagnosis information of patients and the conclusion of ECG examination. The positive patient's ECG examination and preoperative diagnosis information contain the disease information of patients' heart, while the negative patient's preoperative diagnosis and ECG examination conclusion are normal, so the text-based method is very effective for the prediction of heart failure. In order to further integrate the patient's diagnosis information, this paper integrates all the patient's diagnosis information, and carries out the following experiments.    Fig. 13 shows that DPCNN-XGBOOST is the best predictor of heart failure (including weighted average accuracy (98%), weighted average recall rate (98%), F1 score (98%) and ROC value (99%)). Compared with preoperative test data, intraoperative monitoring data and preoperative text attributes, the sensitivity of the model increased by 50%, 31% and 3% respectively. It can be seen from the experimental results that the combination of DPCNN and XGBOOST classifier can improve the prediction of heart failure, and also verify the effectiveness of the proposed model.

Discussion
Heart failure in the perioperative period is one of the most significant causes of postoperative death of patients. At present, most studies only focused on the numerical data of preoperative test or intraoperative monitoring, whose prediction has poor timeliness, low accuracy, and non-fusion of patient heterogeneity data. Thus, in order to make the results of research and analysis more accurate and more convenient to apply to practice, in this paper, we use Deep Pyramid Convolutional Neural Networks (DPCNN) and Extreme Gradient Boosting (XGBOOST) method to model the risk prediction of heart failure after operation based on the preoperative and Intraoperative medical data of patients. The proposed model is verified in four experiments. The results of Experiment 1 show that XGBOOST model has a better classification effect on patients' numerical structural data. The results of Experiment 2 showed that the monitoring data of patients during operation had certain precursory information for the prediction of heart failure. Experiment 3 shows that the text-based data of preoperative examination can better describe the patient's condition. In Experiment 4, the combination of numerical and textual data can further improve the accuracy of heart failure prediction. There are still several ways to improve this work, which is also the direction of our future work. First of all, the word vector used for text feature extraction is based on Tencent's trained word vector. In order to improve the description of patients' condition, we can train medical field word vector based on electronic medical record data for feature extractions. Secondly, the prediction results of the model are still black boxes for doctors. In order to improve the interpretability of the model, it is necessary to further integrate the deep neural network and the tree model, and use the tree model to interpret the results. Finally, the model is only tested in the event of heart failure at present, and we will try to apply

Model comparison
Weighted avg precision Weighted avg recall Weighted avg f1-score Roc the model to other critical diseases (liver failure, renal failure, etc.). In the future, we hope that such methods will be used to provide medical staff with the support to improve decision making for surgical surgeon.

Conclusion
This paper discusses an interesting medical problem. The prediction of heart failure in perioperative patients is a complex process, and whether or not adverse events occur after surgery depends entirely on the doctor experience and judgment, and the prediction accuracy of long-term and experienced doctors is higher, but the prediction accuracy of doctors with short working hours or inexperience is slightly lower. Additionally, the judgment has lag and the evaluation result cannot be applied directly and effectively, which is a serious problem. Therefore, based on the machine learning method, this paper establishes a prediction model of heart failure adverse events in patients to predict the risk of critical disease with the preoperative index of any patient. Firstly, the data of preoperative patients are preprocessed, and the patient data is divided into the structural data of numerical data and the unstructured data of textual data. The numerical data of the patient is constructed by the gradient boosting tree model, and the textual features of the patients are extracted by Deep Pyramid Convolutional Neural Networks of the text-based data. Fusion of textual features and numerical features, and finally through a XGBOOST model to predict the patient heart failure illness.