{'fraction_non_alphanumeric': 0.031358735340865154, 'fraction_numerical': 0.016645904033678424, 'mean_word_length': 4.886837779744595, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards smart diagnostic methods for COVID-19: Review of deep learning for medical imaging

The infectious disease known as COVID-19 has spread dramatically all over the world since December 2019. The fast diagnosis and isolation of infected patients are key factors in slowing down the spread of this virus and better management of the pandemic. Although the CT and X-ray modalities are commonly used for the diagnosis of COVID-19, identifying COVID-19 patients from medical images is a time-consuming and error-prone task. Artificial intelligence has shown to have great potential to speed up and optimize the prognosis and diagnosis process of COVID-19. Herein, we review publications on the application of deep learning (DL) techniques for diagnostics of patients with COVID-19 using CT and X-ray chest images for a period from January 2020 to October 2021. Our review focuses solely on peer-reviewed, well-documented articles. It provides a comprehensive summary of the technical details of models developed in these articles and discusses the challenges in the smart diagnosis of COVID-19 using DL techniques. Based on these challenges, it seems that the effectiveness of the developed models in clinical use needs to be further investigated. This review provides some recommendations to help researchers develop more accurate prediction models.


Introduction
In early December 2019, the first case of COVID-19, the disease caused by the virus Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2), was confirmed by the authorities in Wuhan, China. COVID-19 developed rapidly into a global outbreak and spread all across the world. The World Health Organization (WHO) described the outbreak as a Public Health Emergency of International Concern on 30 January 2020 and declared it a pandemic on 11 March 2020. Globally, as of 6:06pm CEST, 19 October 2022, there have been 623,000,396 confirmed cases of COVID-19, including 6550,033 deaths, reported to WHO ( https://covid19.who.int ).
Although Reverse Transcriptase-Polymerase Chain Reaction (RT-PCR) is the gold standard for diagnosing COVID-19 [1] , the sensitivity of the test is relatively poor, and thus even in patients with a negative RT-PCR result, COVID-19 infection cannot be entirely excluded [ 2 , 3 ]. Therefore, medical imaging, especially chest computed tomography (CT) scan and X-ray, is often used to complement the RT-PCR test to achieve more diagnostic certainty. These imaging techniques have a higher sensitivity than RT-PCR and play a critical role not only in the early diagnosis and treatment of COVID-19 patients but also in monitoring the progress of the disease [ 2 , 3 ]. However, the accuracy of COVID-19 diagnosis using CT and X-ray chest images depends on radiological expertise, and some radiologists may fail to interpret accurately the results of these images, hence leading to reduced sensitivity [ 4 , 5 ].
With recent advances in machine learning techniques, particularly deep learning (DL), and the success of these techniques in medical image processing, scientists and clinicians hope to improve the accuracy of COVID-19 diagnosis by applying deep learning methods to chest medical images. These methods have the potential in providing decision support for clinicians and reducing medical errors. In a short period, we have witnessed a large number of DL models developed for a very broad range of COVID-19-related applications. The purpose of the present review is to give more insight into deep learning applications in the diagnostics of COVID-19. We survey the literature on diagnosing patients infected by COVID-19 using DL techniques for a period from January 2020 to October 2021. Herein, our main focus is on classification and segmentation models proposed based on CT and X-ray chest images. Our review only includes peer-reviewed, well-documented articles and summarizes the technical details of models developed in these articles. We also discuss the challenges of using deep learning for the smart diagnosis of COVID- 19. The challenges show that the performance of the models developed in the reviewed articles is probably optimistic and these models are not of potential for clinical use. We finally provide some recommendations that can help researchers develop more accurate and practical models for the COVID-19 diagnosis.

Data sources and search strategy
This search was intended to address the question: "What deep learning techniques have been developed for the COVID-19 diagnosis using CT and X-ray images? ". For a period from January 2020 to October 2021, we conducted systematic searches of the following on-line databases in order to identify relevant works: IEEE Xplore, ScienceDirect, Springer, PubMed, and Google Scholar.

Search terms
The keywords used for the literature search were "COVID-19; Coronavirus; Diagnosis; Detection; Artificial Intelligence; Machine Learning; Deep Learning; Medical Imaging; CT-Scan; X-ray.'' They were connected using "and'', or "or'' to identify the articles that deal with the diagnosis of COVID-19 using DL techniques on medical images.

Selection criteria
The studies with the following criteria were included in the review: (1) Articles that employed machine learning or deep learning techniques for the COVID-19 diagnosis, (2) Articles that utilized techniques to analyze radiographic images (CT scan, and/or X-ray), (3) Articles that applied classification and segmentation models.
The following exclusion criteria were used to eliminate studies from consideration: (1) Articles that were not in the English language, (2) Articles that were not published in peer-reviewed journals, (3) Review Articles, (4) Articles that did not use ML/DL approaches for diagnosing COVID-19 based on medical images, (5) Articles that did not provide a clear explanation of the implemented model and its results.

Technical background
This review paper aims to survey different DL models developed in the literature for the smart diagnosis of COVID-19 in people with suspected infection. We focus on classification and/or segmentation techniques using CT and X-ray chest images. For this purpose, this section briefly discusses various DL architectures applied in the reviewed articles.

Classification models
In this section, the different deep neural networks for COVID-19 classification based on CT and X-ray chest images are discussed.
Standard CNN . The architecture of convolutional neural networks (CNN) is inspired by human and animal brains. The main advantage of CNN is detecting relevant features automatically and without any human supervision. CNN tries to overcome the overfitting problem by using convolutional layers. Fig. 1 depicts the architecture of standard deep CNN. The different parts of this architecture are described below: • The convolutional layer is the main building block of a CNN and applies a convolution operation to the input. • The pooling layer reduces the dimensions of the feature maps. • The fully connected layer is usually placed at the end of CNN architecture to do the classification task. • The loss function calculates the prediction error. Through network learning, this value is optimized. • Various techniques such as dropout, batch normalization, and data augmentation that are applied for better learning.
AlexNet. AlexNet is one of the earliest and most widely cited deep neural networks and a leading architecture for any object-detection task. It first introduced two new concepts named Local Response Normalization (LRN) and dropout to help the deep neural network learn better [7] . As a normalization layer, LRN implements the idea of lateral inhibition. Dropout is one of the regularization approaches to avoid the overfitting problem. It randomly skips some neurons during training and forces the other neurons in the layer to pick up the slack. The architecture of AlexNet is shown in Fig. 2 .
SqueezeNet. SqueezeNet is a convolutional neural network that utilizes design strategies to reduce the number of weights and is considered a more compact replacement for AlexNet [9] . On the ImageNet dataset, it achieves accuracies comparable to AlexNet while performing 3x faster and containing up to 50x fewer parameters. ImageNet ( http://www.image-net.org ) is a large-scale hierarchical dataset of annotated images for computer vision and machine learning research [10] . Fig. 3 depicts a combined architecture of AlexNet, SqueezeNet, GoogleNet, and MobileNetV2.
VGGNet. VGG is a classical CNN architecture. It consists of a few convolutional layers that apply the RELU activation function [7] . The number of convolution layers differs in various versions of VGG. A single max-pooling layer and three fully connected layers are placed following these activation layers, As shown in Fig. 4 . This architecture uses the Softmax classifier.
GoogleNet. GoogleNet, also called InceptionNet-V1, proposed the novel Inception blocks that improve recognition accuracy. The distinctive features of this architecture (see Fig. 3 ) are the large convolution masks, the less dimensional space, and reduced computational complexity. GoogleNet is deeper with much fewer parameters compared to its ancestors AlexNet and VGG [7] .
MobileNet. The main characteristic of MobileNet (see Fig. 3 ) is using depth-wise separable convolution that consists of two layers named depth-wise convolution and point-wise convolution. Using these layers, MobileNet provides an intensive reduction in model size and decreases computational cost by about 8-9 times compared to standard convolutions. A RELU and batch normalization are placed after both of these layers. MobileNet is simple but efficient for mobile and embedded machine vision applications [13] .
DenseNet . As a type of convolutional neural network, DenseNet utilizes dense connections between convolutional layers where each layer  is connected to every other layer. DenseNet was introduced to overcome the issue of vanishing gradient. With effective usage of feature reuse, the network parameters decrease dramatically [14] . Fig. 5 depicts the architecture of a DenseNet network.
ResNet. The core idea of the Residual network (ResNet) is introducing a technique called skip connection (or Shortcut Connection) that skips some of the layers in the network and feeds the output of a layer as the input to later layers, as shown in Fig. 6 . There are different variants of ResNet architecture composed of a different number of layers. The most popular type of ResNet network is ResNet50 including 49 convolutional layers followed by one fully connected layer [14] .
EfficientNet. The EfficientNet network relies on a compound scaling method that uniformly scales all three dimensions of depth/width/resolution while maintaining a balance between all network dimensions. using the compound scaling technique, the authors scaled the baseline network EfficientNet-B0 to get different variations including EfficientNet-B1 to B7. In comparison to squeeze-andexcitation networks, EfficientNet is 7.7 x smaller and 10 x faster [17] . InceptionNet. An inception network is composed of repeating components referred to as Inception modules. InceptionNet employs the idea of auxiliary classifiers to address the gradient vanishing problem and improve the convergence of very deep networks. It also applies different convolution kernels of various sizes in parallel. This architecture helps to extract similar features in different sizes simultaneously. The popular versions of InceptionNet are InceptionV1 to V3 (as shown in Fig. 8 ) and Inception-ResNet.

Segmentation models
The most common deep learning model developed for image segmentation, U-Net, has been widely utilized for lung region segmentation to diagnose COVID-19 patients: U-NET. U-Net architecture is primarily designed for segmentation applications. Its good performance in segmenting medical images makes it the primary tool for segmentation in this field. The U-Net architecture is an almost symmetrical u-shape network that consists of two paths called decoder and encoder. It is much faster than other segmentation networks, because of its context-based learning [19] . The architecture of U-Net is described in Fig. 9 .

Generative adversarial networks
Generative Adversarial Network (GAN) is an unsupervised approach and a powerful method to produce unseen data samples with the same statistics as the training set. This way, it can overcome overfitting and data shortage problems and improve deep network performance. GAN architecture (as shown in Fig. 10 ) is composed of two sub-models named generator model and discriminator model contesting each other in a  game. The generator model tries to generate new plausible images similar to real data. The discriminator model tries to distinguish between real and fake samples. This process continues until the generator samples become close to the actual input samples [7] .

Results
From all the databases considered for finding relevant works (see Section 2.1 for more information), we identified 15,417 articles that satisfied our search terms, of which, 6209 articles were preprints (not peer-reviewed) and excluded from this review. For duplicate studies (157 articles), we ensured that the latest version of the article was considered. The titles, abstracts, and full texts were screened for relevance and eligibility. We removed 8983 articles that were irrelevant or not documented with enough detail to allow other researchers to reliably reproduce the results. Finally, our investigation retained 68 peerreviewed, well-documented articles for consideration in this review. This section presents all the technical details of DL models developed in these articles for COVID-19 diagnosis in summary and explains the advantages and disadvantages of the used techniques. Table 1 summarizes the reviewed articles and reports informative details for them.
Generally, imaging modalities considered in COVID-19 diagnosis are X-ray, Computed Tomography (CT), and ultrasound. Although ultrasound imaging is a more widely available, cost-effective, safe, and real-time imaging technique [22] , it has a relatively lower sensitivity in comparison to chest CT-Scan [23] and cannot usually detect lesions that are deep and intrapulmonary [ 24 , 25 ]. Since more deep learningbased COVID-19 detection and segmentation works focus on CT and X-ray modalities, we only reviewed articles that have used these modalities. The column Imaging Modality in the table indicates which imaging         modalities have been used as the input to the DL network developed in each article. Chest radiological imaging, such as CT and X-ray, has a vital role in the early diagnosis and treatment of COVID-19 disease [26] . The second column Dataset refers to image datasets collected by or utilized in the reviewed articles. These datasets contain CT and/or Xray images from healthy, COVID-19, and Non-COVID patients. Since COVID-19 is a new disease, there is no dataset of appropriate size for its diagnosis purpose. Therefore, many works combine images from several different publicly available datasets, gather chest images from multiple institutions, or use a combination of their clinical images and public datasets.
The column # Cases Per Class indicates the number of images/patients in each class. In other words, the value of each cell in this column shows which classes have been considered in the corresponding article, and how many images or patients there are in each class. COVID-19 datasets are strongly imbalanced which can impact the model accuracy. Hence, several studies have proposed different class balancing techniques to overcome the data imbalance issue [27] .    [21] .
The column Test Method shows which technique has been used for splitting data into three non-overlapping parts including the Training (Tr) set, Validation (V) set, and Test (Te) set. The division of input data into these three subsets is crucial in the creation of robust prediction models to avoid overfitting, as well as to increase generalization. Overfitting is one of the main issues in the training of machine learning algorithms [28] . It occurs when the training error is low, and the generalization error is high. Although a small training dataset is the main reason for the overfitting problem, the complexity of the model can be another reason [29] . Many works apply a validation set to identify the overfitting problem during the model training by controlling the model complexity [30] . It can help the model determine when to stop the train-ing process. For simplicity, we use two different patterns for reporting the test method. The first pattern is Tr%: V%: Te% which indicates the percent of data assigned to each part, and the second pattern is Tr-V-Te showing the number of data assigned to each part. We use the phrase "Separate DB " for indicating the studies which have utilized separate datasets for training and testing their DL models.
The column Validation Method refers to the validation method used to evaluate the efficiency of DL models. The applied validation methods contain K-fold cross-validation and the holdout method. The K-fold cross-validation technique ensures that every observation from the original dataset has the chance of appearing in the training and test sets [31] . In this approach, at first, the data is randomly split into K equal-sized folds (the K value is usually selected in the range of five to ten depending on the data size). Then, the model is trained on the K-1 folds and validated on the remaining fold. This process is repeated K times, with each of the K folds used exactly once as the test set. This technique is one of the best validation approaches if there are not enough instances to train on [32] . On the other hand, the holdout method partitions data randomly into two sets, training, and test. The size of the test set is typically smaller than that of the training set. Since the method involves a single run, results depend on how the data is split into these two sets [33] . The holdout method is not effective for comparing multiple models and tuning their hyper-parameters. Therefore, another very popular form of this method is usually utilized which splits data into three separate sets including training, validation, and test sets [34] . The validation set which is a holdout subset from the training set is used to tune various hyper-parameters and choose the best performing model. The final model is evaluated by the test set and the found error is considered the generalization error.
The dataset used for the transfer learning process is reported in the column Transfer Learning . The philosophy behind transfer learning is that people can intelligently utilize knowledge learned in the past to solve new problems faster or more efficiently [35] . Transfer learning is an approach in machine learning where knowledge learned in one or more tasks is transferred to improve learning in another related task [36] . The main goal of this technique is to decrease training task time, improve generalization performance [37] , and deal with small input dataset problems [38] . Transfer learning has been widely adopted in many COVID-19 studies to compensate for the scarcity of large-scale public datasets for this disease.
The column Learning Method can contain the values Supervised, Unsupervised, Semi-supervised, and Weakly-supervised, showing which machine learning approach has been used for training the deep neural network. Supervised learning refers to a class of methods that learn on a labeled dataset, where each training sample is a pair of inputs and its desired output [39] . In an unsupervised learning model, in contrast, the algorithm tries to extract features and patterns from unlabeled data. In other words, the algorithm does not receive any feedback from the environment [40] . Clustering, association, and dimensionality reduction are three main tasks in unsupervised learning. A weakly-supervised model utilizes training data with incomplete, inexact, and inaccurate annotation [41] . Finally, semi-supervised learning describes a class of algorithms that seek to learn from both unlabeled and labeled samples typically assumed to be sampled from the same or similar distributions [42] .
Since labeling lesion annotations is time-consuming and impractical for clinicians [43] , Noisy labels exist widely for the segmentation of large-scale 3D medical images. It can be either due to challenges for accurate annotation, such as low contrast, ambiguous boundaries, and complex appearances of the target, or caused by low-cost inaccurate annotations such as annotations provided by non-experts, human-inthe-loop strategies [44] , and some algorithms generating pseudo labels [ 45 , 46 ]. For this reason, some COVID-19 studies have applied weakly supervised or semi-supervised learning frameworks in their segmentation tasks.
The pre-processing functions applied to input images are shown in the column Pre-processing . In image processing, the pre-processing phase aims to process raw input images to improve the quality of images, produce a more understandable format for the algorithm, enhance accuracy, increase the number of data, reduce the processing time, or standardize data acquired from multiple devices [ 47 , 48 ]. There are several important steps in the pre-processing phase, such as data augmentation, histogram equalization, image normalization, morphological operators, segmentation, standardization, etc.
The column Classification categorizes COVID-19 classification models into two main groups: COVID-19 from Non-COVID and COVID-19 from Other Pneumonia. The former indicates a two-class classification where the class Non-COVID includes common pneumonia cases and nonpneumonia cases, and the latter shows a multi-class classification that distinguishes among COVID-19, other types of pneumonia, and healthy cases.
The column Segmentation classifies COVID-19 segmentation models into two categories. The first category, called the Lung-Region-Oriented method [ 47 , 49 , 50 ], is a prerequisite step in COVID-19 applications and separates the whole lung region and lung lobes from other regions in input images. The second category, called the Lung-Lesion-Oriented method [ 44 , 51 ], aims to separate lesions or metal and motion artifacts from the lung regions which is a challenging detection task.
The column DNN , which stands for deep neural network, shows the networks utilized for classification and segmentation tasks. In addition to well-known models, some new DNNs have been also introduced in the reviewed articles to detect COVID-19 infection in CT and X-ray chest images.
The value of the column Classifier indicates the classifier algorithm used in deep learning models for COVID-19 diagnosis purposes. These functions are applied in the last layer of a neural network. Some of the most used classifiers are introduced as follows. The Softmax function [52] normalizes the real values of the input vector into a probability distribution consisting of probabilities. The output values are in the interval (0,1). The support vector machine (SVM) [53] constructs a set of hyperplanes that provide the maximum margin distance to generate the least generalization error. The Sigmoid function [52] , also called the logistic function, is a very popular activation function for artificial neural networks. The input value to this function is transformed into a probability that is between 0 and 1. The Majority Voting classifier [54] merges different classification rules to produce a classifier that is superior to any of the individual ones. It classifies each input value in the class that obtains a large number of votes.
The column Post-processing shows post-processing procedures applied to the output of a deep neural network. Some of the most common post-processing techniques include generating heat-map, heat-map visualization, calculating infection confidence, and GRAD-CAM (Gradientweighted Class Activation Mapping).
Finally, the column Performance Criteria reports the performance of deep learning models developed in the studies. The common performance criteria contain Accuracy (Acc), Sensitivity (Sen), Specificity (Spe), F1-score, Precision (Pre), Recall, and the area under the ROC curve (AUC).
It is worth mentioning that some of the reviewed articles have proposed several DL models for COVID-19 diagnosis purposes or evaluated their models in two-class and multi-class modes. In these cases, for each study, we only provide the details of the model which has obtained the best result that may be the most helpful to researchers.

Discussion
During the recent coronavirus pandemic, RT-PCR frequently produced false-negative findings that increased over time [55] . Due to high false-negative rates, this test sometimes considers COVID-19 patients to be healthy people, which has severe consequences. The probability of a false-negative result on one day a patient has contracted the virus is 100% and falls to 67% over the four days of infection. On day five, the typical time of symptom onset, the chance of a false-negative result is 38% and drops to 20% on day eight. Then it begins to increase again, from 21% on day nine to 66% on day 21 [56] . As a result, an earlier diagnosis tool is necessary to reduce the spread of the disease.
Chest imaging could be a part of COVID-19 diagnosis in suspected or portable COVID-19 patients in the absence of an RT-PCR test, or where the RT-PCR test results are delayed or are negative in the early infection stage in the presence of symptoms indicative of COVID-19 [57] . Benmalek et al. [58] reported the typical imaging features for confirmed COVID-19 pneumonia cases which can help in the early screening and tracking of the disease.
The chest CT scan has been found to have a sensitivity value of 98% and is one of the most precise methods for diagnostics of this disease [59] . Hence, a chest CT may be viewed as a reliable complementary diagnostic measure for RT-PCR to help physicians assess patients more perfectly [18] . The report by Li et al. [60] showed that an initial CT scan performed as early as 5 days after the onset of symptoms would allow for a confident separation of severe and non-severe patients. Lung abnormalities on chest CT peak within 6-13 days after the initial onset of symptoms reaching the highest severity score [61][62][63] . On the other hand, X-ray has a reported sensitivity of 69% for COVID-19 diagnosis [64] . However, compared to CT, it is faster, less harmful, easy to perform, and costs less [65] . The radiographic findings detected on chest X-ray in COIVD-19 patients show the greatest severity on days 5-10 of symptom onset [66] .
The work carried out by Shazia et al. [15] states that applying deep learning techniques to radiological images for novel coronavirus identification has the potential to reduce the workload of medical practitioners and increase the accuracy and efficiency of COVID-19 diagnosis. In what follows, we discuss the reviewed articles on developing DL models for the early diagnosis of COVID-19 using medical images, and attempt to answer these research questions: What are the most commonly used DL techniques for COVID-19 detection using medical images? What challenges may we face during the deep neural network training process on chest images, and how can we cope with them? What are the advantages and disadvantages of each imaging modality in classification and segmentation tasks?

Deep neural networks for COVID-19 diagnosis
A revolution in image processing and machine vision has taken place with the advent of deep learning techniques [120] . Deep neural networks have been used in the domain of image processing to solve difficult problems including image classification, detection, and segmentation [121] . These networks eliminate the need for manual feature extraction. They have introduced the concept of end-to-end learning by receiving annotated images as input and discovering the underlying patterns in each image class automatically [122] . A convolutional neural network can have hundreds of layers where each layer learns to detect different image features. DNNs have also achieved great success in analyzing medical images by providing high accuracy, stability, scalability, and efficiency [123] . They can provide Health professionals with more details about the internal organs and patient's tissues, and help physicians diagnose many types of disease, such as pneumonia, brain injuries, cancer, internal bleeding, and pneumonia, brain injuries, cancer, internal bleeding, etc. [124] . Not only in medical image analysis for the diagnosis of such diseases but also during the recent COVID-19 pandemic, deep learning has proved its success [23] . Gaur et al. [125] give evidence on the successful application of deep learning techniques for COVID-19 infection detection. Their work confirms that deep machine vision models can be implemented in the healthcare sector to screen and detect COVID-19 from chest X-rays. Sethi et al. [126] also utilize different deep CNN architectures to provide doctors with diagnosis recommendations for COVID-19. They show that CNN-based architectures have the potential to diagnose COVID-19 disease using chest X-ray Images.
A variety of deep neural networks developed for COVID-19 diagnosis purposes is shown in Fig. 11 . This figure also shows which classifiers and how many times have been used in each DNN. Due to using different datasets with different sizes and properties, a direct comparison between deep networks considering performance metrics is not possible. However, based on our review, Res-Net and VGG are the most common networks used in classification tasks, and for segmentation tasks, various versions of U-Net deep neural networks have been utilized.
Different experimental studies have proved good results of the stateof-the-art deep architectures VGG and ResNet for challenging recognition and localization problems such as image segmentation [127] . The main advantage of VGG architecture is its good generalization ability in new datasets [128] . In the 2014-ILSVRC competition, VGG became famous, despite being in 2nd place, due to its simplicity, homogenous topology, and increased depth [127] . On the other hand, ResNet can prevent the degradation problem of deep CNNs. This problem occurs when the accuracy gets saturated and quickly degrades with network depth increasing [129] . ResNet has shortcut connections that skip one or more layers and facilitate deep network training without adding extra parameters or computational complexity. ResNet also lets feature maps from the initial layers that usually include fine details easily propagate to the deeper layers [130] .
It is clear also from Fig. 11 that the majority of works have utilized the Softmax classifier in their models. As mentioned before, this classifier represents a probability distribution over class labels, and its output can be directly displayed. Besides, since output values are between 0 and 1, they can be directly fed into any other model without the need for normalization.
It is worth mentioning that using different pre-processing approaches, diverse classifiers, applying the transfer learning method, and other techniques employed for compensating for data shortage could directly affect the results of DNNs. A sufficient number of clinically annotated data is critical for the training step. However, there are very few clinical datasets on COVID-19 which are publicly available and these datasets contain a limited number of COVID-19 cases. Several reviewed studies [ 18 , 21 , 79-85 , 103 , 37 , 45 , 51 , 74-78 ] have applied transfer learning and image preprocessing techniques such as image augmentation or used a combination of different datasets to overcome the COVID-19 data leakage and data imbalance problems and enhance their performance. However, these approaches may lead to high variance estimation of deep learning models' performance during the test phase [131] . Since many of the COVID-19 prediction models are poorly reported, and at high risk of bias and model overfitting [132] , more investigation will be needed to ensure the performance of these models in clinical use. Fig. 12 shows the total number of studies conducted in the field of segmentation and/or classification for diagnosing COVID-19 patients. Based on the results of this figure, most studies have only focused on the classification task. As mentioned before, we categorized COVID-19 classification approaches into two main classes named COVID-19 from Non-COVID, and COVID-19 from Other Pneumonia, which respectively refer to two-class and multi-class classifications. Although the differentiation of COVID-19 from other types of pneumonia is sometimes challenging, an equal number of studies have modeled the COVID-19 diagnosis problem as a two-class and multi-class classification task. Some researchers have also taken advantage of both classification and segmentation techniques to improve model accuracy. These models first extract the ROI (Region of Interest) of the image in the pre-processing phase and then search within the ROI region instead of the whole region. However, most classification works have preferred to process the whole of the image rather than the ROI region. The main reason is that Fig. 11. Number of times that each pair of DNNs and classifiers has been used in reviewed papers for COVID-19 diagnosis. annotating COVID-19 medical images for a segmentation task is difficult and time-consuming for clinicians.

Imaging modalities for COVID-19 diagnosis
The most common imaging modalities for COVID-19 infection identification are CT scan, Lung Ultrasound (LUS), and Chest X-ray radiography. The chest X-ray is low cost, easy to perform, and has low radiation. It also is fast and immediately available for radiologist analysis; the last feature has made it one of the first imaging modalities during the COVID-19 pandemic [133] . Nevertheless, it has limited sensitivity in detecting lung lesions in the early stages of infections [134] . While the Chest CT, being a rapid imaging procedure, may be more sensitive and accurate for the early diagnosis of COVID-19, it exposes patients to more radiation than typical X-rays. Moreover, it is time-consuming for radiologists to diagnose COVID-19 from CT-scan images (about 21.5 min for experienced ones). After each patient, CT scanners will need to be sanitized which is a time-consuming, tedious, and expensive process [ 135 , 136 ].
Compared to the other two modalities, chest ultrasound is low-cost and radiation-free. It can accurately detect the location of objects in real-time and be applied to different lung diseases [137] . Ultrasound has a low false-negative rate in COVID-19 diagnosis [138] . All COVID-19 abnormalities which are visible in CT scan images are also presented on ultrasound images just as clearly [139] . Its accuracy in identifying lung pathologies is better than Chest X-ray, it also is portable, safe, noninvasive, repeatable, and easy to use [134] . However, it shows low sensitivity in comparison to chest X-rays, and it cannot detect deep and intrapulmonary lesions [140] .
In the absence of a CT scan, a chest MRI could be recommended for suspected or confirmed COVID-19 patients. Although MRI has low anatomic resolution and is in danger of inevitable artifacts because of breathing motion, its ability to visualize variations in lung structure is constantly evolving [141] .
Some other imaging techniques have also been utilized for the COVID-19 diagnosis such as Hyperspectral Imaging (HSI), Scanning Electron Microscopy (SEM), and Positron Emission Tomography (PET). HSI is a medical imaging technique that offers noninvasive disease diagnosis. HSI, also known as Imaging Spectroscopy, can capture spectral information for multiple wavelengths at each image pixel [142] . The maximum contrast in hyperspectral imaging is related to the maximum particle concentration [143] . The main disadvantages of hyperspectral imaging are its cost and complexity. To analyze hyperspectral data, fast computers, sensitive detectors, and large data storage capacities are required.
SEM is a powerful tool for infectious disease diagnosis and scans the sample surface morphology by bright focused electron beam emission [144] . It can produce high-resolution and high-quality images that reveal complex and delicate structures. However, it has several cons. In this technique, samples must always be in a vacuum, so this technique cannot be used for live specimens. Electron microscopes are large and need plenty of space in a laboratory. They also are high-sensitive and could be affected by magnetic fields and vibrations of other lab equipment. Artifacts may be present in SEM images and need to be avoided [145] .
PET is a type of nuclear medicine imaging. It applies a small amount of radioactive material to visualize and measure variations in metabolic processes. This non-invasive approach can display very early changes in the cells. Despite the advantages of the PET approach, it is not a routine examination for COVID-19 diagnosis because of its low global accessibility, high inspection cost, complex scanning procedures, and concurrent risks of virus spreading.
Considering the importance and the role of CT and X-ray chest images in diagnosing COVID-19 patients, our review has investigated machine vision-based techniques developed in the literature for the early diagnostics of COVID-19 and other types of pneumonia using these imaging modalities. Fig. 13 illustrates the total number of research works reviewed for each one. It also shows the number of classification and segmentation tasks conducted for each modality. As one can see, most researchers have selected to work on the X-ray modality. Also, the combination of CT-Scan and segmentation in the diagnosis of COVID-19 patients is more common than X-ray and segmentation. The reason is that CT-Scan produces a 3D view of the chest region and allows for a more detailed view compared to X-ray. Most X-ray modality-based studies have focused on classification for COVID-19 diagnosis, which is a less challenging task.

Challenges
Deep learning approaches require a large amount of data to build a robust predictive model. However, COVID-19 is a new infectious disease, and the lack of large-scale labeled data is the main challenge in this area. It might lead to overfitting and reduce the accuracy of DL-based models. To overcome this challenge, a majority of works [ 21 , 37 , 51 , 74-85 ] have combined several public datasets to compensate for the scarcity of well-annotated datasets of appropriate size. However, duplication of images across these datasets is a potential risk during data combination tasks.
Transfer learning is another technique to cope with overfitting and data shortage problems. It is more likely to have a more accurate model by fine-tuning the pre-trained model than training the model from scratch on a small dataset [ 77 , 78 , 146 ]. Some studies [ 18 , 45 , 47 , 49 , 80 , 113 ] have used transfer learning to effectively train models on relatively small labeled datasets in COVID-19 segmentation tasks. However, they have mostly applied deep neural networks pretrained on the ImageNet dataset which includes images that are entirely different from chest images. A small number of segmentation tasks have also used chest images with pseudo labels for the pre-training stage [45] . Preparing noisy labels for the segmentation task is much easier than obtaining clean annotations at the pixel level. Consequently, some research works have attempted to develop a framework to learn from noisy label images [44] .
Class imbalance is another main issue in COVID-19 diagnosis since the number of data related to COVID-19 is much less than other pneumonia diseases. Several works [ 21 , 37 , 67 , 69 , 70 , 100 , 103 ] applied data augmentation methods to compensate for the small COVID-19 datasets and avoid unbalancing data problems. Also, some researchers [ 21 , 75 , 79 , 85 ] used the Generative Adversarial Network (GAN) framework in the preprocessing phase to generate synthetic images to deal with this problem.
Most of the COVID-19 images compress into Non-DICOM formats leading to quality loss. Since the quality of public health information is essential for health monitoring, the lack of image quality remains a challenge in the COVID-19 diagnosis problem. The work carried out by Harmon et al. [97] has reconstructed CT images by super-resolution technique to achieve better accuracy. This technique improves the accuracy of algorithms by making enhanced images with higher contrast.
Generally, lung segmentation can improve the accuracy of a classification task and reduce misdiagnosis [147] . As mentioned before, we grouped the COVID-19 segmentation approaches into two main categories, named the lung-region-oriented methods and the lung-lesionoriented methods. Lesion detection is a challenging task in the medical imaging area as the lesion size is small in comparison to the whole lung size. In addition, the shape of lesions and their texture and location are very diverse. Most works [ 108 , 148-150 ] applied U-Net architecture or Attention-based U-Net to segment lesions and lung regions in COVID-19 automatic detection tasks. Another challenging task is to discriminate COVID-19 from other types of viral pneumonia since COVID-19 is similarly caused by an infective agent. Several DL-based models have been developed in the literature to distinguish between them [ 51 , 68 ].

Recommendations
The awareness and knowledge of the different clinical features of COVID-19 are essential in the early diagnosis and management of the recent pandemic. Some studies were conducted to compare the clinical presentations of COVID-19 patients versus patients infected with other types of pneumonia [151][152][153][154][155] . Researchers found that considering a combination of information such as background and clinical findings of patients, the duration of the symptoms, ancillary imaging findings, and follow-up CT-Scan imaging when needed would be helpful in the differential diagnosis [152] . Based on this, we strongly recommend developing DL-based models that utilize clinical text data, as well as CT and X-ray images to provide a more accurate diagnosis of COVID-19. While many DL-based techniques have been proposed in the literature for detecting COVID-19 patients based on X-ray images, CT-scan images, or both, less work is being done on diagnosis and prediction using clinical text data. Moreover, to the best of our knowledge, there is no research work in the literature focusing on COVID-19 diagnosis based on both medical images and textual data.
As we mentioned before, dealing with small-size datasets is the main challenge in the COVID-19 diagnosis area which can lead to overfitting and reduce the model generalization performance. Researchers may address this problem through various model enhancement and data generation techniques. Since more complex models with more hyperparameters are more overfitting-prone than shallower ones, the adaptation of model complexity to the complexity of data can help researchers overcome the issue of overfitting [156] . Another effective way to have more accurate predictions is combining the results of multiple models, which is called ensemble learning [157] . This approach involves several individual models combined in some way such as weighted averaging, and voting. The ensemble model obtains better generalization performance compared to any of the individual models.
Data imbalance is another main issue in diagnosing COVID-19 patients which can undermine model predictability. The oversampling methods can alleviate both data scarcity and data imbalance problems [158] . One approach to oversampling is generating synthetic samples using augmentation techniques such as GAN.

Conclusion
COVID-19 has spread dramatically in a short time around the world. The SARS-CoV-2 infection causes symptoms from weak to severe in different people, and to date, more than 6.5 million deaths have been reported to WHO ( https://covid19.who.int ). Early diagnosis of COVID-19 can help to prevent the spread of this pneumonia, and save many lives. Although the standard test for COVID-19 diagnosis is RT-PCR, this test is time-consuming, and it is possible to show false-negative results [159] . Therefore, many researchers applied deep learning models to detect COVID-19 in the early stage using X-ray and CT images. Chest imaging has a useful role in the detection and management of COVID-19, even in the early stage of the disease. It helps in suspected patients' identification, Breaking the disease transmission chain, and preventing the further spread of infection [113] . Deep learning approaches can extract rich features from chest images. They are capable of interpreting medical images and distinguishing different types of pneumonia. Physicians can predict COVID-19, estimate its severity, and extract and interpret the infected region by the data extracted from these networks [65] .
During the course of this study, we have undertaken a comprehensive review of some 60 papers that have appeared in the literature on the subject of deep learning-based COVID-19 segmentation and classification using X-ray and CT images. We reported the informative details of each model proposed in the reviewed papers. We also listed the public datasets for diagnosing COVID-19 patients and provided several charts summarizing various techniques used in the different phases of developing a DL-based diagnostic model for COVID-19.
Finally, we discussed the challenges in the research on COVID-19 diagnosis based on medical imaging. We found that the reviewed models are at high risk of bias especially because of the overfitting problem, and their generalization performance is therefore unreliable. Consequently, none of these models are recommended for clinical use. Since the scarcity of large-scale public annotated datasets is one of the main challenges leading to overfitting, the compilation, curating, and sharing of well-annotated datasets is urgently needed in order to develop and validate more accurate models for diagnosing COVID-19.

Funding
None.

Ethical approval
Not required.

Declaration of Competing Interest
None declared.