{'fraction_non_alphanumeric': 0.045478784185267004, 'fraction_numerical': 0.030776262664736802, 'mean_word_length': 4.643809114359415, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
McS-Net: Multi-class Siamese network for severity of COVID-19 infection classification from lung CT scan slices

Worldwide COVID-19 is a highly infectious and rapidly spreading disease in almost all age groups. The Computed Tomography (CT) scans of lungs are found to be accurate for the timely diagnosis of COVID-19 infection. In the proposed work, a deep learning-based P-shot N-ways Siamese network along with prototypical nearest neighbor classifiers is implemented for the classification of COVID-19 infection from lung CT scan slices. For this, a Siamese network with an identical sub-network (weight sharing) is used for image classification with a limited dataset for each class. The feature vectors are obtained from the pre-trained sub-networks having weight sharing. The performance of the proposed methodology is evaluated on the benchmark MosMed dataset having categories zero (healthy control) and numerous COVID-19 infections. The proposed methodology is evaluated on (a) chest CT scans provided by medical hospitals in Moscow, Russia for 1110 patients, and (b) case study of low-dose CT scans of 42 patients provided by Avtaran healthcare in India. The deep learning-based Siamese network (15-shot 5-ways) obtained an accuracy of 98.07%, the sensitivity of 95.66%, specificity of 98.83%, and F1-score of 95.10%. The proposed work outperforms the COVID-19 infection severity classification with limited scans availability for numerous infection categories.


Introduction
In late 2019, the novel Corona Virus Infection Disease  is detected in humans and declared a pandemic by World Health Organization (WHO) on 11 March 2020 [1]. The COVID-19 belongs to the virus family of Severe Acute Respiratory Syndrome-Corona Virus 2 (SARS-CoV2) and Middle East Respiratory Syndrome-Corona Virus (MERS-CoV). In the end of July 2021, globally there were 1,94,080,019 active cases of coronavirus, 4,162,304 deaths reported, and 3, 696,135,440 people were vaccinated [2]. In the year 2021, researchers have claimed the presence of numerous mutants of COVID-19 such as B.1.1.7, B.1.351, and P.1, etc. [3,4]. The mutation of coronavirus is having a huge impact on the transmission scale of infection in humans, the efficacy of testing kits, vaccination, and treatment procedures [5].
In the year 2021, the highest number of coronavirus cases are found in India with an average of 0.35 million cases per day in April [6,7]. The number of cases rapidly increased due to community transmission and the average time duration of the The chest X-ray of COVID-19 suspected patients provides a preliminary diagnosis of the dense tissues and the impact of infection in the lungs in a 2-D view [9]. However, in chest CT scans, patches due to infection are visible in 3-D views (axial, coronal, sagittal). Therefore, to speed up the COVID-19 infection detection process, radiologists prefer chest CT scans over X-rays. Doctors differentiate the healthy control CT scan from the coronavirus infection patient's CT scan measurements based on consolidation, pleural effusion, and Ground Glass Opacity's (GGOs) in the lungs as shown in Fig. 1. Further, the measurements of these patches in the left and right lung predict the severity of coronavirus infection. However, manual annotation of the patches due to COVID-19 infection is an error-prone and tedious task.
In recent times, researchers focused on the development of a future prediction model for COVID-19 cases variants, and improvements in the available COVID-19 measurement equipment as well [10,11]. In [12], low dose CT scan measurements are suggested for the severity of COVID-19 infection detection in mild COVID-19 infected patients along with children and pregnant women CT scans. The COVID-19 cases are predicted using time series models such as Eigenvalue Decomposition of Hankel Matrix (EVDHM) along with the Auto-regressive Integrated Moving Average (ARIMA) model [13], Composite Monte Carlo (CMC) simulation [14]. To monitor the lung condition of critical covid patients on ventilators, a data acquisition system, i.e., Electrical Impedance Tomography (EIT) system is proposed [15].
The major challenges associated with COVID-19 infection severity measurement using CT scan-based modality are (a) limited dataset is available with expert annotation, (b) multi-class classification of the level of COVID-19 infection is not given, etc. Thus few-shot learning frame-work is introduced for classification on a small sample dataset or no labeled dataset [16]. The Few-shot learning approach include metric-learning and metalearning-based techniques. In the computer vision field, a meta learning-based approach is introduced for face identification [17], signature verification [18], and a variety of plant leaves classification tasks, etc. [19]. In medical field, numerous disease detection task with limited dataset availability is resolved used few-shot learning based Siamese networks [20][21][22]. The Meta-COVID networks (2-ways, 10-shot) are implemented for binary classification of COVID-19 and healthy X-ray images with an accuracy of 96.5%, specificity of 98.4%, and precision of 98% [23].
The main contributions of the proposed work are as follows: 1. The deep learning encoder-based Siamese network is proposed for the multi-class classification of COVID-19 infection from lung CT scan slices.
2. The P-shot N-ways (N denotes the number of classes, and P denotes the images per class) based meta-learning framework evaluated using contrastive loss.
3. The Gradient-weighted Class Activation Mapping (Grad-CAM) is used to visualize the multi-class infection features learned by the proposed McS-Net.
4. The proposed methodology is implemented on the Indian dataset of suspected COVID-19 patients with numerous infection levels.
5. Performance evaluation parameters such as sensitivity, specificity, accuracy, F1-score, precision, etc. are obtained for the considered datasets with numerous variations in the encoder network and some samples per class prove the superiority of the proposed framework.
The rest of the article is organized as follows: Section 2 presents the related work; Section 3 puts forth the brief details of the COVID-19 based CT scan dataset. Section 4 illustrates the materials and methods; Section 5 presents the experimentation and discussion. Section 6 concludes the proposed work along with the future scope.

Related work
Various Artificial Intelligence (AI) based techniques are implemented by researchers to auto-mate the COVID-19 infection detection, classification, segmentation, and future prediction task. In [24], X-ray modality is used for multi-class classification of COVID-19 infection with an accuracy of 97.72 ± 0.95% for severe infection, 86.90 ± 3.20% for moderate level, and 61.80 ± 5.49% for mild COVID-19 infection. The Internet of Things (IoT) based pre-trained deep learning model detected the COVID-19 with a sensitivity of 96.73% [25]. The IoT-based approach using a blood sample, X-ray images, body temperature of the covid suspected patient obtained the specificity of 97.95%. Gen-ProtoPNet based deep learning model obtained the classification accuracy of 87.27% on chest X-rays for three classes (normal, COVID-19, and Pneumonia) [26]. The ResNet-SVM obtained a classification accuracy of 93% and sensitivity of 88% on X-ray images of COVID-19 patients [27]. The Genetic Algorithm (GA) with specific hyperparameter optimization technique is used for binary classification of COVID-19 infection [28]. However, the technique takes more than 25 h to get the best results due to the specific optimization parameters requirement. Table 1 put forth the brief details of state-of-the-art techniques available for COVID-19 diagnosis using radiographic images.
In [34], combination of transfer learning and shallow learningbased approach is used to detect the severity of the COVID-19 infection. The proposed method implemented on cloud-based server detects the binary class of COVID-19 infection severity. Hybrid system of the Sugeno fuzzy integral with ensemble of four pre-trained deep learning model (Squeeze Net v1.1, wide ResNet-50-2, VGG-11, and Google Net) is used for binary classification of COVID-19 [35]. In [36], customized DenseNet201-based models are used for COVID-19 diagnosis with an accuracy of 98.18%, a precision of 97.76%, and the specificity of 98.17%. Further, joint classification and regression method is used for binary classification of severity of COVID-19 infection [37]. In [38], fully automatic approach obtained a global accuracy value of 97.06%. In [39], ResNet50 with majority voting-based model is implemented for COVID-19 classification. The fine-tuning based ResNet50 model obtained Area Under the Curve of 0.90 and 3D-ResNet50 model obtained 0.67 of AUC.
In [40], two-stage deep learning based framework is proposed to discriminate between COVID-19 infection from normal pneumonia using chest CT scans. Transfer learning-based approach is implemented on chest X-ray images to detect COVID-19 [41]. For dataset-1, MobileNet+ support vector machine (SVM) classifier obtained a F1-score of 98.5% and DenseNet201+multi-layer perceptron (MLP) obtained a F1-score of 95.6% for dataset-2. The  [42] technique with augmentations of the classification branch in the RoI layer obtained sensitivity of 93.55% for COVID and 99.37% of F1-score. In [43], 5656 X-ray images of each healthy and covid patient is classified using VGG network. The proposed model obtains an accuracy of 96.41%, sensitivity of 96.6%, and the specificity of 96.2%. The explainable classification module (ECM) is proposed for COVID-19 classification using chest CT scan with an accuracy of 84.83% [44]. The hybrid (SpaSA and CNN) approach is implemented on chest CT scans of COVID 19 dataset [45]. The SpaSA algorithm is used to optimize the different Convolution Neural Network (CNN) and transfer learning hyperparameters. The proposed work obtained an accuracy of 99.74% on binary class (14,486 images) and 98% accuracy on three class (17,104 images) classification framework. The 2D-Empirical wavelet transform (EWT) along-with pretrained transfer learning-based approach (DenseNet121) is used for binary classification of chest CT scans [46]. The TL+DenseNet121 approach obtained an accuracy of 85.50%, F1score of 85.28%, and AUC of 96.60%. In [47], a Parallel Quantum-Inspired Self-supervised Network (PQIS-Net) for automatic segmentation of lung CT images from hybrid dataset. The CNN based model is proposed for multi-class classification of chest CT scans into normal, COVID-19, and pneumonia. The proposed work obtained an average accuracy of 98% [48].
From the state-of-the-art, it can be inferred that chest CT scanbased COVID-19 diagnosis can identify the coronavirus infection with an accuracy of > 90% in comparison to X-rays or antigenbased tests [49]. The COVID-19 infection diagnosis using deep learning-based approach requires a large set of samples for each class and the model needs to be retrained for variation in the sample in a class. Also, the training on a small dataset may lead to overfitting. Further, the disadvantages of the state-of-the-art deep learning-based techniques for COVID-19 infection diagnosis using chest CT scan include (a) multiple levels of severity of the COVID-19 infection are not clear, (b) the COVID-19 infection is difficult to discriminate from certain pneumonia and lung infections, and (c) numerous techniques failed to differentiate the mutations of coronavirus.

Dataset
The important rationale of the proposed methodology is the dataset used for COVID-19 infection severity classification. This section discusses the details of the dataset used based on joint analysis of expert radiologist. In the proposed work, the multiclass classification framework for COVID-19 infection is inspired by the Coronavirus disease 2019 (COVID-19) Reporting and Data System (CO-RADS) [50] and CT severity scores. The CO-RADS score range is based on the patches available in the left lung and right lung [51,52]. The proposed deep learning-based Siamese networks are evaluated on a chest CT dataset with annotations provided by joint analysis of Expert-1 (E1) and Expert-2 (E2). Fig. 2 put forth the annotation details suggested by the experts for the considered dataset. The considered chest CT scans are provided by medical hospitals in Moscow, Russia for 1110 patients within the age of 18-97 years (duration: 1st of March to the 25th of April 2020) [53]. The chest CT scans with no lung infection are named as zero or healthy control. The chest CT scans are provided in NIFTI format [54]. The SMOTE based slice selection method is used to address the class imbalance issue with a considered dataset [55]. For class zero (healthy control), the first 15 axial CT scan slices are considered. Further for the suspected COVID-19 infected patients, the 15 slices with infection patches are considered by the joint decision of experts, i.e., E1∩E2. However, only 2 patients' CT scans are available for class CT-4, thus all the slices are considered.

Methodology
This section discusses the detailed hierarchy of the proposed methodology, i.e., pre-processing of the dataset, brief architecture of pre-trained sub-network, contrastive loss function, and nearest neighbor classifier. The schematic architecture of the proposed methodology is shown in Fig. 3.

Pre-processing
In the considered dataset, chest CT scans are available in NIFTI format with a size of 512 × 512 and an average of 36 slices per patient. The input image dataset pre-processing is required  to make the input dataset compatible with the proposed training network. The pre-processing pipeline includes the following steps: selection of axial slices, resizing operation, normalization, and contrast enhancement. To reduce the storage space issue and computational capacity, the input images are adjusted to the size of 256 × 256. Consider the input image dataset is X k = [(I 1 , H 1 ),(I 2 , H 2 ), . . . , (I n , H n )], here, I k is input images and H k is respective labels for k = (1, 2, . . . , n). The input images with pixels 'i' and 'j' are normalized to the range of [0, 1] and represented as I norm (i, j) in (1).
The contrast of the CT scan slices is enhanced using a histogram equalization approach. For histogram equalization for the normalized input image, i.e., I norm (k), consider 't 0 ' as its cumulative histogram, 'tc' is the cumulative sum of the histogram. The histogram equalization is based on the transformation to map the grayscale to their new values. The monotonic transformation 'M t ' is minimized using tc (M t (r)) -t 0 for all intensities r. Thus, Ic 1 , Ic 2 , . . . , Ic n are the pre-processed CT scan images with corresponding labels.

Pre-trained sub-networks
In the P-shot, M-ways, based learning framework, for each infection class, a maximum of 15 images are used for training.  Thus, pre-trained deep learning networks trained for 1000 classes classification on ImageNet dataset are considered as the base encoder for the considered dataset [56,57]. As the depth of layer increases, the number of trainable parameters increases and that leads to complexity for metric-based learning space. Thus, to reduce the time complexity and metric space parameters ResNet-18 based pre-trained deep learning model is used as a base encoder. Fig. 4 put the architecture details of the base encoder network, i.e, ResNet18 for the proposed work. The ResNet18 with skip connection residual network is inspired by the VGG-19 architecture. The customized weight initializer function 'He' for CNN layers with ReLU layer is calculated with mean value zero and standard deviation, i.e., S d is defined using 2. The scale of ReLU layer is represented by 'b' and 'Fs' is the filter size and 'C ' is number of channels.
The sub-networks in Siamese networks are having two different inputs and prediction scores are obtained based on the similarity of the encoded features as shown in Fig. 5. The Siamese batch pairs are obtained using Algorithm 1. The Euclidean distance is obtained between the obtained embedded vectors and the loss function is used to update the training model.

Contrastive loss function
The generalized loss function is used to evaluate the prediction performance of the trained model. The loss value increases if the predicted value 'y p ' is different from the true label 'y t '. The contrastive loss functions are based on similar and dissimilar pair images. More similar feature embeddings are obtained for the same classes. The Euclidean Distance (ED) metric for the embedded feature values, i.e, F 1 and F 2, is obtained using (3). The 0 is an optimum value used to eliminate the undefined Euclidean distance measurement variables and set them to 1e-6.
Further, the loss function for similar 'L s ' and dissimilar pairs 'L d ' is calculated using (4), (5).

Prototypical nearest neighbor classification
The prototypical classification framework is based on embedded query points and finding the nearest class prototype as shown in Fig. 4. In the proposed P-shot M-ways classification framework, consider the support set with labeled sample S n = {(x 1 , y 1 ), . . . , (x k , y k )}, k = {1, 2, . . . , n}. The prototype of each class is computed in S dimensional space such that J p ∈ R s and embedding function is f ϕ : R D −→ R s as defined in (7).
The distribution over classes for a query point β with learnable parameters ε using (8). The prototype learning is based on minimizing the probability given in (9).

System requirements
The deep learning-based Siamese networks used 'Adam' optimizer, the initial learning rate is 1e−4, gradient learning factor set to 0.9, mini-batch size is 4, the activation function is ReLU, output feature vector size is 1024D, and weight initializer is 'He'. For the pre-trained deep learning-based encoder networks, the bias factor is set to 10. The maximum epoch is set to 50 with an early stopping criterion. The experiment is conducted on MAT-LAB R2021b software installed on Intel core i7 10th Generation processor (64 bit) with NVIDIA GeForce RTX2700 having 8 GB capacity and 16 GB RAM capacity.

Performance evaluation
To evaluate the performance of the proposed work, numerous performance evaluation parameters are considered such as Accuracy, sensitivity, specificity, AUC, and F1-score. The generalized confusion matrix A for the multi-class classification framework is given 10.
µ 4 = (Pr EHI + Pr HI + Pr II + Pr LI +Pr Z ) P (24) To evaluate the robustness of the proposed work, the performance parameters are evaluated for the individual classes as well and given in Table 3. It can be inferred that less accuracy and sensitivity is obtained for 'LI' class, less specificity and precision is obtained for 'II' class. Also, F1-score is 90% and MCC is < 90% for both 'LI' and 'II' classes.

Grad-CAM based infection visualization
The Grad-CAM technique is used to visualize the strongest activation features map learned by the proposed model. The Grad-CAM is based on the linear combination of weights of channel activation maps with an applied ReLU layer. It evaluates the network prediction using differential outputs (class scores) of the gradients. The activation function ReLU provides the feature map of the region of interest, i.e., (severity of infection) in the specified class. In a proposed multi-class infection classification task, consider the score of class is represented by 'S r ' for output 'Y S r ', and feature maps are denoted by 'f '. The obtained score Map (:, :, pc) corresponding to categorical label 'pc' is the gradient of the final classification score for the particular class, w.r.t. each feature in the feature layer. The obtained Grad-Map is up-sampled to the size of 224 × 224 size, i.e., input image size. Fig. 6 shows the obtained Grad-MAP overlapped on respective input images to visualize the severity of infection in lungs CT scans of suspected COVID-19 patients.

Case study on Indian patients
The chest CT scans of 42 COVID suspected patients are collected in April 2021 for the patient (age 21-85 years). The low dose CT scan measurements are obtained on the multi-detector 16 slice CT Aquilion model from Toshiba. The high voltage circuit generates 60 kW power for a 7.5 MHU X-ray tube. The exposure time is 0.35 s for partial scans and 0.50 s for each full scan measurement. The length of the detector along the z-axis is set to 38 mm with 896 channels or columns across 40 rows. The slice thickness range varies from 16 × 0.5 mm to 16 × 1 mm.
The average total time taken by the technician for chest CT slices measurements is 10 min for setup creation.
The CT scans are provided in DICOM 3.0 format having size dimensions 512 × 512 and an average of 150 frames are given for each patient. The input images are pre-processed to make them compatible with the proposed methodology. In the Indian dataset, for each patient 15 slices are considered. Further, Fig. 7 presents the performance evaluation details of the proposed work on the Indian dataset. Based on demographic region, an average recovery time period for COVID-19 patients was 14 days which is reduced to 7 days depending on COVID-19 infection severity [58]. In the proposed work, the chest CT scan of the dataset is collected for 30 days. However, the COVID-19 infection severity is detected for chest CT scan of an average 10 day period of COVID-19 detection to quarantine period. The proposed work obtained a precision of 94.0%, specificity of 98.50%, the sensitivity of 94.54%, F1 score of 93.32%, and MCC is 92.08%.

Ablation study
The proposed methodology is investigated for the following queries: (1) How would different pre-trained CNN models affect the performance of multi-class classification of Siamese based network? (2) What is the impact of variation in P-shots on performance evaluation parameters of ResNet18 base encoder. Table 4 puts forth the brief details of performance comparative analysis of the Siamese-based sub-networks on the considered dataset. It can be inferred that > 90% accuracy is achieved with the VGG19, DarkNets, and ResNet18. However, DarkNets and VGG-19 have higher trainable parameters that increase the complexity of parameters in metric space for similarity index. Thus, ResNet-18 based pre-trained model is a suitable encoder for the proposed work. The proposed ResNet18 encoder performance is evaluated with variation in N-ways and P-shots-based multi-class classification framework. Table 5 puts forth the details of global performance parameters with 5 classes and a P-shots learning framework. It is concluded that 5 class 15 shot training networks provide the optimum performance on the considered dataset. Table 6 put forth the comparative analysis of the proposed methodology with other existing techniques available in literature on the considered MosMed dataset with 5 ways and 15 shots classification framework. The proposed methodology detects the  [20]. The COVID-19 infection severity is classified with an accuracy of 96.20% in 12 s per scan using DenseNet [59].

Limitations and future scope
Earlier we have worked on binary classification of chest CT scans of COVID19 patients from normal patients using deep learning-based techniques [63,32]. For this, numerous statistical data augmentation techniques are implemented. But the earlier reported work is not able to detect the COVID-19 infection severity on smaller dataset. The proposed McS-Nets addresses the issue of COVID-19 infection severity classification with limited data availability in each class. It provides the accurate prediction of the disease in mean time with the minimum number of training samples. However, the proposed work is having certain limitations associated with it: (a) performance parameters are < 90% for the infection level with less than 25% infection or few GGO is visible, (b) it is costly as multiple CT scans are required for follow-up patients (average 2-3 scans during 14 days of tenure of COVID-19 infection first detected).
In future work, the proposed technique is to be implemented in a real-time environment for COVID-19 infection diagnosis. The proposed framework is based on image similarity; thus, techniques need to be explored on various variants of COVID-19 infection. Further, the proposed methodology will be implemented on the following conditions: (a) different variants of COVID-19 infections, (b) blood samples and other clinical parameters will also be used for COVID-19 infection identification. To make the proposed technique more effective, contrastive learning-based segmentation is to be explored. Also, a cloud computing-based framework is to be used for wider clinical applications.

Conclusions
The proposed work addresses the COVID-19 severity infection diagnosis using chest CT scans with limited samples available from each class. The COVID-19 infection in lungs is categorized as: 1. Zero -0% infection (CT-0), 2. Low -< 25% infection (CT-1), 3. Intermediate -25%-50% infection (CT-2), 4. High -50%-75% infection (CT-3), 5. Extremely high infection -> 75% infection (CT-4). The proposed work learned different severity of infection features in the lung area in comparison to the best performing deep learning-based encoder network. In the proposed 15-shot 5-ways McS-Net an average of > 95% of accuracy, sensitivity, and specificity is obtained on testing data. The Grad-CAM-based functionality is used to investigate the infection map learned by the proposed networks. Further, the proposed work is evaluated on the Indian dataset with limited CT scan slices in each class and obtains average accuracy of 97.06%, a sensitivity of 94.54%. Thus, the proposed technique is useful in the infection severity identification on the follow-up low dose chest CT scan of the patients. Table 6 Comparative analysis of the proposed techniques with the exiting state-of-the-art methods on considered MosMed dataset for multi-class classification (5 ways 15 shot).

Declaration of competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.