{'fraction_non_alphanumeric': 0.033074975059473564, 'fraction_numerical': 0.01799554907528202, 'mean_word_length': 4.704311665572336, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Covid‐19 detection from radiographs by feature‐reinforced ensemble learning

Summary The coronavirus (Covid‐19) epidemic continues to have a negative influence on the global population's well‐being and health. Scientists in many fields around the world are working non‐stop to find a solution to the prevention of this epidemic. In the field of computer science, this struggle is supported by studies on especially the analysis of X‐ray and CT images with artificial intelligence. In this study, two different ensemble learning models, including deep learning and a combination of machine learning methods, are presented for the detection of SARS‐CoV‐2 infection from X‐ray images. The main purpose of this study is to increase the classification ability of Residual Convolutional Neural Network (ResCNN), which is used as a deep learning method, with the assist of machine learning algorithms and extracted features from images. The proposed models were validated on a total of 5228 chest X‐ray images categorized as Normal, Pneumonia, and Covid‐19. The images in the dataset were sized in four different ways, 32 × 32, 64 × 64, 128 × 128, and 256 × 256, in order to analyze the validity of the proposed models in more detail. These four datasets were partitioned with the 10‐fold cross‐validation technique and converted into a total of 40 training and test data. Both proposed models use features derived from the ResCNN as the basis and test a certain number of machine learning algorithms with a majority voting technique by dividing them into subsets. In the architecture of the second model, it combines the features extracted from the Local Binary Patterns (LBP) and Histogram of Oriented Gradients (HOG) methods in addition to the features obtained from the ResCNN. It has been seen that the classification ability of both proposed models is better than the ResCNN in the experiments. In particular, the second model gives a similar classification score even though it is tested with images four‐times smaller (e.g., 32 × 32 vs. 128 × 128) than those used in the ResCNN. This shows that the model can give ideal results with lower computational cost.


INTRODUCTION
The novel coronavirus disease (Covid-19) is a virus that was initially found on January 13, 2020, as a result of study undertaken in a group of individuals who acquired respiratory symptoms (fever, cough, and shortness of breath) in late December 2019 in Wuhan, China. 1 The pandemic was first discovered in people who worked in the seafood and livestock markets in this area. 2,3 Later, it expanded from person to person and to other cities in Hubei province, particularly Wuhan, as well as other provinces of China and other nations around the world. 4 Coronaviruses are a broad group of viruses that can infect both animals and people. Several coronaviruses have been linked to respiratory infections in humans, ranging from the common cold to more serious illnesses including Middle East Respiratory Syndrome (MERS) and Severe Acute Respiratory Syndrome (SARS). 5 The SAR-CoV-2 virus causes novel coronavirus illness. Although it has been reported that there may be asymptomatic cases in the symptoms of the novel coronavirus disease, the rate of these is not known. 6 Fever, cough, and shortness of breath are the most prevalent symptoms. 7 Pneumonia, severe respiratory failure, renal failure, and death may occur in extreme cases. Covid-19 virus is transmitted by coughing and sneezing of sick individuals and inhaling droplets scattered around the environment. The virus can be transmitted to the eyes, face, nose, and mouth without washing the hands after contacting surfaces infected with respiratory particles. Contact with eyes, nose or mouth with dirty hands is risky. 8 According to the information available thus far on Covid-19 infection, some people are at a higher risk of becoming ill and having severe symptoms. Accordingly, 80% of the cases have mild disease and 20% are treated in hospital conditions. 9 People with medical conditions such as heart disease, hypertension, diabetes, cancer, and chronic respiratory disease; those over the age of 60, and healthcare workers are most affected by the disease. 10 Machine learning algorithms can be incorporated to build an automatic computer-aided diagnostic (CAD) system to struggle the Covid-19 pandemic. 11 A significant number of clinical and radiological research have been published in this field, reporting various radiographic imaging results as well as the epidemiology of Covid-19. 12 Many deep-learning models, such as convolutional neural networks (CNN), [13][14][15][16] transfer learning models, 17,18 generative adversarial networks (GAN), 19,20 and others, have also been used to automatically analyze diseases diagnosed by radiological methods. Covid-19 is identified utilizing chest X-rays in studies with multiple or binary classifications. Raw image data is used in certain research, whereas feature extraction techniques is used in others. The number of samples in the datasets employed in investigations varies as well. Research in this area has shown that CNNs are the most popular classifiers for corona virus detection. Ai et al. 21 to evaluate performance of chest computed tomography (CT) with use of the reverse-transcription polymerase chain reaction (RT-PCR) in the diagnosis of Covid-19, 1014 patients who underwent both chest CT and RT-PCR tests in Wuhan, China were included in their study. As a result of statistical analyzes with SPSS, they reported that chest CT has a high sensitivity for the diagnosis of Covid-19 and therefore can be considered as the primary tool for corona virus detection in epidemic areas.
Some of these studies are as follows: Waheed et al. 22 developed a method based on the Auxiliary Classifier Generative Adversarial Network (ACGAN), called CovidGAN, to produce synthetic chest X-ray images for the dataset containing a limited number of X-ray images used in their studies and to increase the success of CNN in detecting Covid- 19. In their experimental studies, they reported that they achieved 85% classification accuracy with the use of CNN alone, and this rate increased to 95% when they added synthetic images produced by CovidGAN. Ozturk et al. 14 proposed two deep learning-based models to detect and classify Covid-19 cases from X-ray images. The first is the binary classifier, which labels "Covid" and "No-findings," and the second is the multiclass classifier, which labels "Covid," "No-findings," and "Pneumonia." As a result of their experimental studies, they reported that the proposed methods had an accuracy of 87.02% and 98.08%, respectively. Ismael andŞengür, 23 in their study for corona virus detection, used a dataset containing 380 chest X-ray images, 180 of which were "Covid" and 200 were "Healthy." They used five pre-trained CNN models (ResNet and VGG variants) for feature extraction and the SVM classifier for feature classification. They reported that the ResNet50 + SVM model had the highest classification accuracy with 94.7% among all the results they obtained in their experimental studies. Hussain et al. 24 proposed a 22-layers CNN model they named CoroDet for detection of Covid-19 using CT and chest X-ray images.
In their experimental study, they reported achieved an accuracy of 99.1% with binary classification labeled as "Covid" and "Normal," 94.2% with triple classification labeled as "Covid," "Pneumonia," and "Normal," 91.2% with quadruple classification labeled as "Covid," "Viral Pneumonia," "Bacterial Pneumonia," and "Normal." Chandra et al. 25 proposed a Covid screening system, called ACoS, which detects healthy, suspected, and corona virus-infected patients using radiomic texture descriptors extracted from X-ray images. The ACoS has a two-stage classification approach using a majority vote-based classifier set of five supervised learning methods. They used 2088 and 258 chest X-ray images for training-test and validation, respectively. According to the classification results obtained, they achieved an accuracy rate of 98% in the first phase and 91% in the second phase. Barstugan et al. 26 to classify X-ray images for the diagnosis of corona virus using five different feature extraction methods that are Local Directional Patterns (LDP), Discrete Wavelet Transform (DWT), Grey-Level Size Zone Matrix (GLSZM), Grey-Level Run Length Matrix (GLRLM), and Grey-Level Cooccurrence Matrix (GLCM). These features were classified by SVM using 2-fold, 5-fold, and 10-fold cross validation techniques. They reported that the highest classification accuracy rate was 99.68% using the 10-fold cross-validation for features extracted from the GLSZM. Narin et al. 27 used five different pre-trained CNN classifiers to detect Covid-19 cases using chest X-ray images. In their experimental study, they used 5-fold cross validation on three different datasets and reported the highest accuracy rate between 96% and 99.7% with ResNet50.
The contributions of this study are listed as follows: 1. A novel ensemble learning model is presented for diagnosing Covid-19 infection from chest X-ray images. The proposed model differs from traditional machine learning and deep learning methods in terms of its architecture.
2. The proposed study trains a certain number of machine learning algorithms using features extracted from Residual Convolutional Neural Network (ResCNN), Local Binary Patterns (LBP), and Histogram of Oriented Gradients (HOG). It separates and tests them into subsets and returns the subset with the highest classification accuracy based on majority voting as the solution.
3. By combining the power of deep learning and machine learning algorithms, it provides higher performance and is very practical to implement due to its automatic feature extraction. 4. Applicable to all classification problems involving images, as it does not need problem-specific parameters.
The rest of paper is organized as follows: The Section 2 contains technical details and sample images about the dataset used in the experimental studies. In Section 3, there are detailed explanations about the LBP and HOG, which are feature descriptor methods. In Section 4, proposed classification models for detection of Covid-19 infection from chest X-ray images are explained in detail.
In addition, technical details about the ResCNN architecture used for comparison in experimental studies are given. In Section 5, the proposed models validated by experimental study on the dataset and classification performances of the ResCNN, PM1, and PM2 were compared. In addition, performance of the PM2 according to class labels is documented in detail. In the last section, this paper is summarized.

DATASET
The dataset is a collection of 5228 Covid-19 chest X-ray images (radiographies) available from Kaggle, a subsidiary of Google LLC. Specifically, the images are divided into three classes as shown in Table 1; the total number of normal cases is 1626, bacterial infection cases (Pneumonia) 1802 and viral infection cases (Covid-19) 1800 confirmed data. This dataset is publicly available on Kaggle. 28 Figure 1 shows all three classes of radiographic X-rays.

FEATURE EXTRACTION
Feature extraction plays an important role in pattern recognition and classification. 29 Appropriate and distinctive features can effectively present a variety of content in an image, while also providing a solid basis for ultimate classification. In this study, in addition to the features obtained from Residual CNN, features were extracted by applying LBP and HOG techniques to X-ray images in the dataset. Then, all these features were brought together and used in training and testing the proposed method.

Local binary patterns
LBP is a type of visual descriptor (or texture operator) used in computer vision for classification. 30,31 The LBP is calculated by thresholding intensity of circularly symmetric neighboring pixels with the intensity of the center pixel, in a local region of an image, as shown in Figure 2. In this respect, it can be seen as a unifying approach to the typically diverse statistical and structural concepts of texture analysis. In the calculation process, the neighbors with positive density differences are set to one and the others to zero. After this operation, the resulting binary (0 and 1) values are converted to decimal numbers. When the pixels surrounding the center are all black or all white, this image region is flat (i.e., there are no features).
Consecutive groups of black or white pixels in the local region are considered "uniform" patterns that can be interpreted as vertices or edges. If the pixels alternate between black and white pixels, the pattern is considered "non-uniform." The LBP has become a prominent technique in a variety of applications due to its computational simplicity and distinctive power. The most important property of the LBP in real-world applications can be said to be resistant to monotonic grayscale changes caused by illumination variations in images. In Equation (1), the notation (P, R) is used for pixel neighborhoods that means P sampling points on a circle of radius of R. Let I to be a grayscale image, and I(x, y) to be a gray value at (x, y) in I. The LBP at (x, y) is therefore defined as: where, I (x i , y i ) is intensity of P equally spaced pixels in a circle with radius R centered on pixel (x, y), and (x i , y i ) is the location of neighbors given by X.
If the neighbors are not in the center of the pixels, interpolation should be used to approximate their intensities. Equation (2) shows how to calculate the LBP P,R value using (P, R), as in Figure 3.

F I G U R E 2 Result samples of the LBP relative to the central pixel
Since LBP produces 2P different output values, the LBP value of each pixel in an image can be calculated and generate a 2P-bin histogram as the image descriptor.
The original LBP provides invariance to any monotonic transformation and scaling of the gray scale. Each rotation invariant LBP is given a unique ID, which is defined as stated in Equation (3), to achieve rotation invariance.

Histogram of oriented gradients
The histogram of directed gradients (HOG) is a feature descriptor used in machine vision and image processing for object detection and image classification processes. 32 The HOG identifier focuses on the structure or shape of an object. 33 It is better than any edge descriptor as it uses magnitude as well as gradient angle to calculate features. Generates histograms for regions of the image using the gradient size and orientations. The purpose of this method is to express the image as local histograms. These histograms contain the number of repeats at certain offset values of the gradient orientations of the image. This simple but effective feature extraction method has gained intense interest in the literature due to its successful applications. Especially in CNN and computer vision, it has found a wide area of use on different problems. 34 In the HOG method, the gradient values for each pixel in the image are first calculated using Equation (4).
where, g x and g y are the horizontal and vertical gradient, respectively, and I(x, y) is the pixel value at position (x, y) of the image. The gradient magnitude and direction are then calculated using Equation (5).
After the gradient of each pixel is obtained, the gradient matrix is divided into k × k cells and descriptive blocks (C m,n ) are created. By default, the value of k is usually taken as 8. In the next step, an n-bins histogram (H) in the range of 0-180 • is calculated for each block, as shown in Figure 4.
Here n is used to specify the angle range, which is usually 9. Thus, the step size becomes Δ = 180∕n, that is, 20 • . The values to be given to the hth and (h + 1)th divisions, respectively, for each cell in a block are calculated by Equation (6). Here h = ⌊ ⌋ and = ( ∕Δ ) − 0.5.
Because a block contains 64 different values, calculations are made for all 64 magnitude and gradient values. When the histogram calculation for all blocks is finished, four blocks from the 9-bins histogram matrix are combined to form a new 2 × 2 grid box (B) as in Figure 5.
For all 4 cells in a grid box, combining all the 9-bin histograms for each block to form 36 feature vectors, . In the last step, the B i values for each grid box are normalized with the L2 norm as in Equation (7).
F I G U R E 4 Representation of a 9-bins histogram where, is a small value added to the square of B i to avoid division by zero. The received code value is 1E − 05. In this way, the necessary calculations are made for each defined grid boxes in the horizontal and vertical directions of the gradient matrix, and the HOG features are obtained.

PROPOSED METHODS
In this study, two ensemble learning models with different architectures are presented. The common features of both models are as follows: In order to increase classification accuracy of the dataset containing chest X-ray images, the capabilities of deep learning and machine learning algorithms are combined. The features of chest X-ray images were extracted by deep learning, and machine learning methods were used as classifiers. The  In the third and final phase, majority voting in Equation (9) is performed for the predictions of machine learning algorithms with combination in Equation (8). Let S = {LDA, SVM, NB, DT, kNN} be a set representing machine learning algorithms that can be used to solve a classification problem.
Accordingly, r-element subsets (i.e., combinations without repetition) of the S: The number of machine learning algorithms used in this study is five (|S| = 5) and the value of r is determined as 3. Thus, the number of 3-element subsets (M) of the S is C(|S|, r) = 10. Where C represents r-combinations of the S. The majority voting for the ensemble decision is: where, is actual class label of a feature (or input) vector, f P ( m i,j ) is the function that returns predicted class label of the m i,j and is the Kronecker Delta (KD) function in Equation (10).
By performing the above processing steps, 10 ensemble learnings (M), each consisting of three different machine learning algorithms (m i ), are tested and used as classifier with the highest KD-score. Figure 8 shows the architecture of the proposed model #2 (PM2). The difference of this model compared to PM1 is that classifiers are supported by the LBP and HOG techniques.
In order to achieve higher classification accuracy, features were extracted from pre-processed chest X-ray images with the LBP and HOG techniques. Since the number of features obtained with HOG is quite large, NCA was applied to select some of them. The selection constraint was determined as the feature weights being greater than the relative convergence tolerance in the gradient norm. In the final stage, the classifiers were trained by combining the features obtained from LBP, HOG (selected by the NCA), and FCL of the ResCNN.

EXPERIMENTAL RESULTS
Experimental studies consist of four stages. In the first stage, the images in the chest X-ray dataset were sized in four different ways as 32 × 32, 64 × 64, 128 × 128, and 256 × 256 pixels. All images were then gray scale transformed and normalized, and pre-processing was completed. After this process, the image dataset was partitioned with 10-fold cross-validation to measure the more realistic performance of the classifiers. The cross-validation is used to test model's ability to predict new sample that was not used in predicting it, in order to flag problems like selection bias or overfitting. In addition, it is used to give an insight on how the model will generalize to a different dataset. In the classical approach, the accuracy of the model can be subjective as a result of manually partitioning the training and test data in certain proportions (e.g., 80% and 20%). For these reasons, it is aimed to obtain more objective and realistic results by partitioning dataset with the cross-validation technique.
Thus, 40 training/test data in different data sizes and combinations were applied to all classifiers. In the second stage, training and testing results were obtained for each cross-validated dataset using the ResCNN deep learning algorithm. In addition, 1000 features from ResCNN's FCL were recorded for use in the proposed methods. In the third stage, five different machine learning algorithms were trained with the features obtained from ResCNN, called PM1 as shown in Figure 7 and tested using the majority voting technique with triple combinations. In the last stage, in addition to PM1, called PM2 as in Figure 8, the features obtained from the pre-processed images using the LBP and HOG techniques are combined with the features obtained from the ResCNN, and the training and testing of machine learning algorithms as in the PM1. The results were analyzed in detail.
All data in the dataset are labeled as Covid-19 (COV), Normal (NOR), and Pneumonia (PNE). All experiments were conducted using MATLAB version 2020b on a Core i7 CPU machine with 8 GB of RAM and 3.30 GHz. Table 3 shows the training and testing results of the ResCNN on the 10-fold cross-validation dataset. According to the results obtained, it is seen that the training and test scores gradually increase as the image sizes increase. Naturally, information loss occurs with the reduction of image sizes. In mean classification accuracies, the highest score was 96.77% for images in original image size, while the lowest score was 97.93% for 32 × 32 pixel images.  Figure 9 shows the mean classification accuracies of the PM1, ResCNN and machine learning algorithms according to different image sizes. In Table 4, the classification results of the same methods with 10-fold cross-validation are given in detail.
As a result of the experimental studies, it was seen that PM1 slightly improved the classification accuracy compared to ResCNN for all image sizes. The machine learning combination that won the most in the majority voting was the trio of the k-NN, LDA, and SVM.  To measure the performance of the PM2 according to the class labels, four measurements, namely accuracy, sensitivity, specificity and precision, were recorded. According to the results obtained, it was observed that the COV label was stable with an accuracy rate of 98.7% in all measurements.
When we look at the precision rates, it is 99% in 256 × 256 pixel images and 98% in the others. Precision value is especially important when the cost of False Positive (FP) estimation is high. For example, if the patient who should be labeled as NOR in the radiological diagnosis is labeled as COV or PNE (FP), then it will weaken the reliability of the system by saying that it is a sign of infection for healthy people. For this reason, a high precision  Figure 11. In addition, AUC (Area Under the ROC Curve) scores were calculated separately for each class in sub figures. The AUC C , AUC N and AUC P labels on this figure represent AUC-scores of Covid19, Normal, and Pneumonia, respectively. The AUC-scores for each model increase in proportion to the image sizes. Therefore, the lowest scores are on 32 × 32 pixel images, and the highest scores are on 256 × 256 pixel images. The most striking point in these figures is that AUC-scores of the ResCNN with 128 × 128 pixel images, the PM2 was able to obtain with 32 × 32 pixel images.

DISCUSSION AND CONCLUSION
The CNNs can extract features of objects by scanning images with convolution kernels at a significantly reduced computational cost. It also increases the resistance of network to spatial variations thanks to the pooling layer. The features obtained by the CNN tend to be abstract that are difficult to explain but complete the image classification or recognition as an end-to-end model. Although deep learning networks such as CNN have reduced the computational cost to a certain extent with convolution kernels, the processing time increases in proportion to the image sizes. Reducing the image size naturally causes loss of information. As a result, the classification score of the model tends to decrease. We can clearly see this in the test accuracies of the same dataset at different image sizes in Table 13.
In learning algorithms is eliminated by using the features derived from the ResCNN. Thus, by combining the predictions of many machine learning algorithms with majority voting techniques, it has been transformed into an ensemble learning model, which can achieve higher classification accuracy.
If we consider the performance of the proposed models in terms of both time cost and classification accuracy; the classification accuracy on the ResCNN's 128 × 128 pixel dataset shown in Table 13 is 95.76% and the time spent on training the model is 24,143 s (6 h 42 min), as can be seen in Table 14. The PM2, with its 32 × 32 pixel dataset, achieved a score very close to the classification accuracy of the ResCNN's 128 × 128 pixel dataset. Looking at the total time spent for this in shorter time.
Although the classification performance of the PM1 was slightly better than the ResCNN, it lagged behind the PM2. In the PM1 model, machine learning algorithms are trained using features obtained entirely from ResCNN. Although the majority voting technique slightly improved the results, it did not achieve a higher classification score as in the PM2. The reason why the PM2 gives better results than the PM1 is undoubtedly the inclusion of the HOG and LBP extracted features. Thus, by further enriching the features produced by the ResCNN, it strengthened the distinctive ability of the model. The PM2 was applied to the chest X-ray image dataset and the obtained classification results proved to be significantly successful. In addition, it has been proven by experimental studies that reduces the running time cost between four and five times in order to achieve classification accuracy of the ResCNN. It is hoped that the proposed method will improve Covid-19 detection and lead to more reliable radiology systems.