{'fraction_non_alphanumeric': 0.031720741599073, 'fraction_numerical': 0.024430281962147547, 'mean_word_length': 4.622421281216069, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Convolutional Neural Networks in Multi-Class Classification of Medical Data

We report applications of Convolutional Neural Networks (CNN) to multi-classification classification of a large medical data set. We discuss in detail how changes in the CNN model and the data pre-processing impact the classification results. In the end, we introduce an ensemble model that consists of both deep learning (CNN) and shallow learning models (Gradient Boosting). The method achieves Accuracy of 64.93, the highest three-class classification accuracy we achieved in this study. Our results also show that CNN and the ensemble consistently obtain a higher Recall than Precision. The highest Recall is 68.87, whereas the highest Precision is 65.04.


Introduction
Deep Learning algorithms are the state-of-art AI techniques that use self-optimization through learning (LeCun et.al, 2015;O'Shea & Nash, 2015). Deep Learning algorithms became major investigative tools in life and social science, with health science and medical science significantly benefitted from their applications (Holzinger et al, 2017). Deep Learning methods are based on neural networks, a representation of human neurons in the form of abstract data structure. Whereas a basic classical neural network consists of three layers (the input layer, the hidden layer, and the output layer), the term "deep" often refers to multiple hidden layers. The learning results depend on many factors, including the network architecture, optimization algorithms, the selected loss function. Data preprocessing also plays an important role in the obtained results. This includes feature selection and construction, instance sample selection, including over-and under-sampling of imbalanced data.
In this work, we report applications of Convolutional Neural Networks (CNN) to multi-classification classification of a large medical data set 1 . The current work continues our previous work on explainable multi-class classification of medical data (Hu and Sokolova, 2020). As in the previous work, we gather empirical evidence on Diabetes 130-US hospitals for years 1999-2008 dataset. The set and its features were introduced by Strack et.al. (2014) and deposited to the UCI Machine Learning Repository. The dataset has 100000 instances and 55 features. 2 The data processing, including noise removal and knowledge-based feature construction, has been presented and discussed in detail in (Hu & Sokolova, 2020).
We work on multi-classification of the patient's readmission, which resulted in classification of three classes: 0, <30, or > 30 days. This is a novel task that provides more insights into patient readmission prospects than the binary classification of patients' readmission. The results of the three-class classification can lead to finer-grained patient-centric and privacy protection analysis of the data.
We discuss in detail how changes in the CNN model and the data pre-processing impact the classification results. To select the best model, we compute Macro F-score, Precision, Recall, and Accuracy (Sokolova & Lapalme, 2009). In the end, we introduce an ensemble model that consists of both deep learning (CNN) and shallow learning models (Gradient Boosting). The method achieves Accuracy of 64.93%, the highest three-class classification accuracy we achieved in this study. Our results also show that CNN and the ensemble consistently obtain a higher Recall than Precision. The highest Recall is 68.87, whereas the highest Precision is 65.04.
In this work, we use software supported by Scikit-Learn and PyTorch. Data analysis and processing libraries include Pandas, Matplotlib, and NumPy. All coding and model training is done by using CPU I7-9700K and GTX 1060.

Preliminaries
Convolutional Neural Networks (CNN) consist of convolutional layers, pooling layers and fullyconnected layers (O'Shea &Nash, 2015). The key factors of CNN applications can be listed as follows: local connections, shared weights, pooling, and the use of the layers. The idea behind location connections is based on certain data sets, e.g., images and signal sequences, being highly correlated within the same location; thus, the feature maps may exhibit the same motif. For each round of CNN application, the input data are broken into several feature maps. Each of the feature maps can be treated as the input of a layer, with a convolutional layer followed by a pooling layer taking in the inputs to process the features. Different from regular neural networks -where each neuron is associated with one weight and connected to the next -each convolutional layer shares an array of weights called filter. The weighted sum from the filter and the convolutional layer is then passed into a non-linearity function and then fed into the pooling layer.
The pooling layer, used after the convolutional layer, merges the feature maps with similar features. (The feature maps that are close to each other are semantically identical, therefore the original inputs can be condensed into a smaller set of features.) A common CNN architecture has a few stacked convolutional layers and pooling layers. Their outputs are forwarded to the fully-connected layers. In the end, the weights in filters will be trained by the backpropagation. In practice, there will be another parameter called out_channels which determines the number of outputs at each round of convolution; the parameter helps the networks to generate multiple outputs based on different choices of weights instead of training based on one weight.

Vanilla CNN
We start with a vanilla CNN network, that incorporates CNNs as its layers. The network consists of 6 layers, with 2 layers of CNN, 1 layer of max pooling, and 3 forward layers. Architecture is shown in Figure 1; our configuration parameters are shown in Table 1.

Figure 1, Architecture of the vanilla CNN
We use cross-entropy loss function as our criterion to decide on the predicted label. The crossentropy loss function in Pytorch combines the negative log likelihood loss and log softmax function, and outputs a numeric probability value on different classes. We use the largest value as our predicted label. Note that the negative log function is more sensitive when the confidence of a classified class is low. In the other words, the negative log function will give more penalties when the confidence of classification is low; this can be one of the advantages of using the Cross-Entropy loss function that consists of negative log-likelihood and log softmax function. The formula of Cross-Entropy Loss function used in this report is illustrated below We opted for Adaptive Moment Estimation (Adam) (Kingma et.al., 2014) as our initial optimizer, since it performs well when dealing with large datasets. The Adam algorithm is outlined as follows: for each parameter, the optimizer will first take the exponential average of the square of gradients and use that value along with the learning rate to define the step size on the weights of the neural networks. In the last step, it adds the modification of the weights to the original weights.
To make sure our training set is unbiased towards the class distribution, we use 10-fold crossvalidation with a stratified shuffle. The vanilla CNN has a learning rate of 0.0001, trained on 64 batches and 10 epochs. To further adjust our vanilla model, we will perform parameter tuning on it. Since we found the learning rate and batches have an impact on accuracy, we will consider epochs, learning rate, and batches as parameters to tune. For each configuration, we will perform 10-fold cross-validation and take the average accuracy for each configuration to compare. Total training time for this step took around 16 hours to finish. We will test each combination of configuration, with epochs value 10,50, learning rate 1e-05, 1e-04, 0.001, 0.01, and batch size 16,32,64. Table 1 reports on Accuracy, Macro Recall, Precision, and F-score. The highest obtained Accuracy is 62.82%. We make a few adjustments to our vanilla CNN. We hypothesize that a deeper network may result in better accuracy, thus we add CNN to the vanilla architecture ( Figure 2). Then we set the window size to 4, to give more information to the multinetwork player in the front. At last, we will add dropout layers to each linear layer in the front; dropout layer will randomly stop update the weights for a few random weights, we set the threshold to be 0.2 in this case. The best obtained Accuracy is 62.91%.

Feature Selection
We investigate the impact of univariate feature selection on the classification performance.
We first perform Chi-square statistical test on the features and the target label. For each feature we will using a Chi-square test to find if there is dependency with the target, and then rank the features according to their score. We use the SelectionKBest method from Scikit-Learn. We select the top 16,32, 58 features for CNN, Gradient Boosting, and Random Forest. The latter two algorithms have achieved the best classification in (Hu & Sokolova, 2020). Features are normalized between 0 to 1 to perform this test since the negative value is not permitted here. However, the Chi-square feature selection does not improve on the accuracy of classification. We drop 16 features from further tests as they provided for the lower accuracy.
We then use Pearson correlation to compute the correlation between each feature and the target, then rank their score and pick the top 20 and 58 features to train the dataset. Features with zero mean value are excluded from this test. In this case, Gradient Boosting has an insignificant increase from 64.75% to 64.83%.
For the ANOVA test (F-value test), we propose null hypnosis for each feature and the target label, then find F-value, rank the features, and train the dataset. We pick the top 20 and 58 features to perform the test. Features with zero variance are excluded from this test. In this case, there is an observable decrease for CNN when 20 features are used. All the results are summarized in Table 2. We report in detail CNN results after the feature selection. All tests are done with 10-fold stratified cross validation on CNN. We also investigate the impact of the 23 medication features (Appendix 1) on the CNN performance; the features had been shown to improve Macro Recall of the patient readmission classification (Hu & Sokolova, 2020).

Oversampling and under sampling methods
To balance the original Diabetes data set, we have been using SMOTE are our oversampling method (Hu & Sokolova, 2020). We performed a few other oversampling methods: ADASYN, Random oversampling, SVMSMOTE, and BorderSMOTE, and an undersampling NearMiss. We adjusted our dataset using these four oversampling methods and found that ADASYN performs the best with an accuracy of 64.40%.

Method
With 23  As shown in the table above, ADASYN provided for the best accuracy among all the oversampling and an under-sampling method. Thus, the following sections will be based on the oversampling method of ADASYN. Our benchmark accuracy is established as 64.41%.

CNN 2.0 adjustment
We work with CNN 2.0 presented in Fig. 2.
Step 1: We explore whether a more complex network might boost up the accuracy. In the new architecture, the last three layers contain neurons 2560, 1280, and 3, respectively. The ad-hoc layer parameters do not improve on the CNN 2.0 performance. The results are given in Table 5.

Feature selection
With 23   Step 2: Next, we are going to explore the impact of the kernel size on the accuracy; from the current 4, we decrease it to 1 and 2 and increase it to 6. As we can observe from the tests ( Step 3 To further explore the impact of kernel size, we made a deeper and more complex networks, as shown by We now use both filter size 3 and filter size 5 layers and combine them to pipe into the original model. The purpose is to use both layers with filter size 3 and 5 to check if the mixed filter size can help to improve accuracy. As we can see the batch size does not have a big impact on the accuracy. Our hypothesis that a larger batch size could let the model learn more examples at a time, thus tuning the weight based on more factors, works when the batch size is small. However, this approach does not work when the batch size > 64, since the accuracy is decreased.

Model
Our current network is using Adam as our optimizer. On this step, we compared its performance with AdaBelief (Zhuang et.al., 2020) and Stochastic Gradient Descent. Both optimizers performer worse than Adam.
From the reported results, we observe that Recall is consistently higher than Precision. With its better Recall results, CNN 2.0 correctly identifies approx. 68% -68.37 % patients of the appropriate class.

Analysis on predicted values
As we have seen from the previous results, our accuracy is bottlenecked around 64%. To overcome the bottleneck, we analyze the class evaluation on each class by using the best model we have build. This is CNN 2.0 built on oversampling method ADASYN ( We take the class evaluations from our best model with 64.41% Accuracy. As we can observe from the table, the Recall class 1 is around 98.45% which means we correctly labeled 98% of the instances in class 1. We can also see that the Precision of class 1 is also higher than the precision of class 2 and 0. This situation happens to all 10-fold cross-validation confusion matrices, as well as the confusion matrix on Gradient Boosting and Random Forest (Hu and Sokolova, 2020). In another word, our model performs well in classifying class 1.
To identify issues with the 0 and 2 class classification, we first assess the feature impact on the 0 and 2 classes classification. We have used Chi-square, Correlation, and ANOVA to analyze the feature importance for the classification. There are 10 common features that the three feature selection methods pick among the top 15 important features. We are going to use them in a further analysis. Table 11 lists their means and standard variations in respect to the three classes.

Feature Name
Class 0  There are two featuresother type (Discharge) and number_impatient -that have a large discrepancy on the mean values in respect to the classes. Other features exhibit similar mean values on the three classes. Thus, the features in isolation do not provide enough information for the classification assessment.
Our next step is to investigate how well the algorithm differentiates between classes 0 and 2. In the data, class 0 has 52337 instances, i.e. approx. 53% of the dataset, class 1 has 34649 instances, i.e.35% of the dataset, class 2 has 11066 instances, i.e., 11% of the dataset. For the binary classification on class 0 and class 2, we randomly pick 11066 instances of class 2 first, then using NearMiss to under-sample class 0 with respect to class 2. After under-sampling, we have 11066 instances of class 2 and class 0 (the data set A). Then we used Gradient Boosting to perform binary classify class 0 and class 2 by using 10-fold crossvalidation. 4 The obtained accuracy is 79.31%. Although the accuracy is high, we note that class 2 is randomly picked, class 0 is under-sampled using NearMiss with respect to the class 2.
This high accuracy will not retain if we randomly pick both class 0 and class 2, when Accuracy drops to 55%. Again, the high accuracy will not retain if we use a balanced dataset without under sampling (the data set B) when Accuracy drops to 68%. In this case, no under-sampling methods is used, thus the accuracy we have is an ordinary prediction by the model. This shows that under-sampling technique helped us to achieve the high accuracy in this binary classification. Noted that we did not pick all the instances of class 2 since we think that the accuracy is enough for us to progress. Further analysis can be done using different sampling techniques.

The Gradient Boosting and CNN ensemble
We make use of the performance improvement we achieved above. We combine Gradient Boosting and CNN 2.0 and employ the results of Sec 4.1.
We use an oversampled (ADASYN) dataset, then train CNN on it, accept classification of class 1, and leave classification on classes 0 and 2 to Gradient Boosting. The dataset is generated as the data set B in the previous section and then we perform oversampling technique ADASYN on the dataset. The total dataset: 106880 instances, divided among three classes.  Table 12 The confusion matrix across 10-fold validation by CNN The confusion matrix above shows the confusion matrix across 10-fold cross-validation. In total, we have 97+25888+263 = 26248 prediction on class 1. We accept all the classification results for class 1. We then remove 26248 instances of class 1 from the data set. Total dataset: 80632 instances divided among two classes. Those 80632 instances have been classified by Gradient Boosting. The results are shown in Table 13.  Table 13 Confusion matrix on binary classification by Gradient Boosting In total, from the original 106880 instances, CNN has correctly classified 25888 instances on class 1, followed by Gradient Boosting that correctly classified 43514 instances on classes 0 and 2. Thus, our accuracy is 64.94%. Compare to the best result we have by using CNN-ADASYN (Table 4, 64.41%), we improved Accuracy on > 0.5%.

Previous Work
Several works used Deep Learning algorithms to classify the patient re-admission stays. We listed results obtained on the same data set. Note that all the previous work has been done for binary classification. Bhuvan et.al (2016), performed an additional cost analysis on patient's readmission and the binary prediction on if the patient will be readmitted in 30 days or would not. They concluded that Random Forest and Neural Networks are the best models with an area of 0.65 and 0.654 by using under precision-recall curve. Hammoudeh et.al. (2018) binary classified would the patient be readmitted in 30 days or would not. The researchers used CNN and achieved an accuracy of 92%. They have applied an additional Znorm to the binary encoded features, removed around 70,000 duplicate instances, and applied early stopping technique. Li et.al (2018) achieved an accuracy of 88% for the binary classification of the readmission days. The team employed CFDL (Collaborative Filtering-enhanced Deep Learning) algorithm. Avram et.al. (2020) correctly identified 82 % of the patients with diabetes. The authors used Deep Neural Networks (DNN).
Different from the listed work, we proposed a novel ensemble model that consists of a Convolutional Neural Networks and Gradient Boosting. Whereas the previous research has focused on the binary classification on readmission in 30 days, we worked on three-class readmission classification: 0, < 30, and > 30 days.

Conclusions and Future Work
In this work, we have reported application of Convolutional Neural Networks to multi-class classification of a large medical data set. Specifically, we investigated the impact of batch size, learning rate, and epoch by parameter tuning, and analyzed why CNN reached a bottleneck approx. 64.41%. We have proposed an ensemble model that consists of CNN and Gradient Boosting. The ensemble overcomes the accuracy bottleneck and improves classification performance with Accuracy of 64.93%, using 10-fold cross-validation.
In addition, we presented the results of different oversampling techniques. In our task, ADASYN outperforms the other methods. By comparing the results obtained by three optimizers, we have shown that Adam outperformed the other optimizers.
For future work on the ensemble model, we could apply NearMiss on each fold of the 10-fold cross-validation, perform binary classification with "0 days" and ">30 days", and then apply CNN on the entire fold for "<30 days". We could also migrate the binary classification to CNN by using the binary cross-entropy function. Another idea is that we could turn the multi-classification problem into three binary classification tasks, then analyze the results and label the test data with the most voted value. Also, RNN deserves more attention, as its parameter tuning has minor influence on accuracy around ~2%.
, Simple RNN architecture We will test our RNN by different epochs and test if the accuracy is sensitive to the number of epochs. We are expecting the higher epochs leads to higher accuracy since more epochs will stabilize the weights in the networks.

Adjusted RNN accuracy
We made a change to the number of epochs after we observed that when training on epoch 50, the loss value tends to decrease at the end of epochs. Thus we decided to train on more epochs. As we can see the accuracy did increase, but it still not better than 64.4%