{'fraction_non_alphanumeric': 0.030475471291027265, 'fraction_numerical': 0.02386795760403463, 'mean_word_length': 4.950890707751565, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
COVID-19 detection in cough, breath and speech using deep transfer learning and bottleneck features

We present an experimental investigation into the effectiveness of transfer learning and bottleneck feature extraction in detecting COVID-19 from audio recordings of cough, breath and speech. This type of screening is non-contact, does not require specialist medical expertise or laboratory facilities and can be deployed on inexpensive consumer hardware such as a smartphone. We use datasets that contain cough, sneeze, speech and other noises, but do not contain COVID-19 labels, to pre-train three deep neural networks: a CNN, an LSTM and a Resnet50. These pre-trained networks are subsequently either fine-tuned using smaller datasets of coughing with COVID-19 labels in the process of transfer learning, or are used as bottleneck feature extractors. Results show that a Resnet50 classifier trained by this transfer learning process delivers optimal or near-optimal performance across all datasets achieving areas under the receiver operating characteristic (ROC AUC) of 0.98, 0.94 and 0.92 respectively for all three sound classes: coughs, breaths and speech. This indicates that coughs carry the strongest COVID-19 signature, followed by breath and speech. Our results also show that applying transfer learning and extracting bottleneck features using the larger datasets without COVID-19 labels led not only to improved performance, but also to a marked reduction in the standard deviation of the classifier AUCs measured over the outer folds during nested cross-validation, indicating better generalisation. We conclude that deep transfer learning and bottleneck feature extraction can improve COVID-19 cough, breath and speech audio classification, yielding automatic COVID-19 detection with a better and more consistent overall performance.


Introduction
COVID-19 (COrona VIrus Disease of 2019) was declared a global pandemic on February 11, 2020 by the World Health Organisation (WHO). Caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), this disease affects the respiratory system and includes symptoms like fatigue, dry cough, shortness of breath, joint pain, muscle pain, gastrointestinal symptoms and loss of smell or taste [1,2]. Due to its effect on the vascular endothelium, the acute respiratory distress syndrome can originate from either the gas or vascular side of the alveolus which becomes visible in a chest x-ray or CT scan for COVID-19 patients [3,4]. Among the patients infected with SARS-CoV-2, between 5% and 20% are admitted to ICU and their mortality rate varies between 26% and 62% [5]. Medical lab tests are available to diagnose COVID-19 by analysis of exhaled breaths [6]. This technique was reported to achieve an accuracy of 93% when considering a group of 28 COVID-19 positive and 12 COVID-19 negative patients [7]. Related work using a group of 25 COVID-19 positive and 65 negative patients achieved an area under the ROC curve (AUC) of 0.87 [8].
Previously, machine learning algorithms have been applied to detect COVID-19 using image analysis. For example, COVID-19 was detected from computed tomography (CT) images using a Resnet50 architecture with 96.23% accuracy in [9]. The same architecture also detected pneumonia due to COVID-19 with an accuracy of 96.7% [10] and COVID-19 from x-ray images with an accuracy of 96.30% [11].
The automatic analysis of cough audio for COVID-19 detection has also received attention. Coughing is a predominant symptom of many lung ailments and its effect on the respiratory system varies [12,13]. Lung disease can cause the glottis to behave differently and the airway to be either restricted or obstructed and this can influence the acoustics of the vocal audio such as cough, breath and speech [14,15]. This raises the prospect of identifying the coughing audio associated with a particular respiratory disease such as COVID-19 [16,17]. Researchers have found that a simple binary machine learning classifier can distinguish between healthy and COVID-19 respiratory audio, such as coughs gathered from crowdsourced data, with AUC higher than 0.8 [18]. Improved performance was achieved using a convolutional neural network (CNN) for cough and breath audio, achieving an AUC of 0.846 [19].
In our previous work, we have also found that automatic COVID-19 detection is possible on the basis of the acoustic cough signal [20]. Here we extend this work firstly by considering whether breath and speech audio can also be used effectively for COVID-19 detection. Secondly, since the COVID-19 datasets at our disposal are comparatively small, we apply transfer learning and extract bottleneck features to take advantage of other datasets that do not include COVID-19 labels. To do this, we use both publicly available and our own datasets to pretrain three deep neural network (DNN) architectures: a CNN, a long short-term memory (LSTM) and a 50-layer residual based architecture (Resnet50). For subsequent COVID-19 classifier evaluation, the Coswara dataset [21], the Interspeech Computational Paralinguistics ChallengE (ComParE) dataset [22] and the Sarcos dataset [20] are used. We report further evidence of accurate discrimination using all three audio classes and conclude that vocal audio including coughing, breathing and speech are all affected by the condition of the lungs to an extent that they carry acoustic information that can be used by machine learning classifiers to detect signatures of COVID- 19. We find that the application of transfer learning enables the classifiers to perform more accurately, exhibit a greater robustness and reduce the tendency of over-fitting. Sections 2 and Section 3 summarise the datasets used for experimentation and the primary feature extraction process. Section 4 describes the transfer learning process and Section 5 explains the bottleneck feature extraction process. Section 6 presents the experimental setup, including the cross-validated hyperparameter optimisation and classifier evaluation process. Experimental results are presented in Section 7 and discussed in Section 8. Finally, Section 9 summarises and concludes this study.
2. Data 2.1. Datasets without COVID-19 labels for pre-training Audio data with COVID-19 labels remain scarce which limits classifier training. We have therefore made use of five additional datasets without COVID-19 labels for pre-training. These datasets contain recordings of coughing, sneezing, speech and non-vocal audio. The first three datasets (TASK, Brooklyn and Wallacedene) were compiled by ourselves as part of research projects concerning cough monitoring and cough classification. The last two (Google Audio Set & Freesound and Librispeech) were compiled from publicly available data. Since all five datasets were compiled before the start of the COVID-19 pandemic, they are unlikely to contain data from COVID-19 positive subjects. All datasets used for pre-training include manual annotations.

TASK dataset
This corpus consists of spontaneous coughing audio collected at a small tuberculosis (TB) clinic near Cape Town, South Africa [23]. The dataset contains 6000 cough audio by patients undergoing TB treatment and 11393 non-cough audio such as laughter, doors opening and objects moving. This data was intended for the development of cough detection algorithms and the recordings were made in a multi-ward environment using a smartphone with an external microphone.

Brooklyn dataset
This dataset contains 746 voluntary coughs by 38 subjects compiled for the development of cough audio classification systems [24]. Audio recording took place in a controlled indoor booth, using a Rode M3 microphone and an audio field recorder.

Wallacedene dataset
This dataset consists of 1358 voluntary coughs by 51 patients, also compiled for the development of cough audio classification [25]. In this case, audio recording took place in an outdoor booth located at a busy primary healthcare clinic. The recording was performed using a Rode M1 microphone and an audio field recorder. This data has more environmental noise and therefore a poorer signal-to-noise ratio than the Brooklyn dataset.

Google Audio Set & Freesound
The Google Audio Set dataset contains excerpts from 1.8 million Youtube videos that have been manually labelled according to an ontology of 632 audio event categories [26]. The Freesound audio database is a collection of tagged sounds uploaded by contributors from around the world [27]. In both datasets, the audio recordings were contributed by many different individuals under widely varying recording conditions and noise levels. From these two datasets, we have compiled a collection of 3098 coughing audio, 1013 sneezing audio, 2326 speech excerpts and 1027 other non-vocal audio such as engine noise, running water and restaurant chatter. Previously, this dataset was used for the development of cough detection algorithms [28].

LibriSpeech
As a source of speech audio data, we have selected utterances by 28 male and 28 female speakers from the freely available Lib-riSpeech corpus [29]. These recordings contain very little noise and the large size of the corpus allowed easy gender balancing.
2.1.6. Summary of data used for pre-training In total, the data described above includes 11202 cough events (2.45 hours of audio). It also includes 2.91 hours of speech from both male and female participants and 2.98 hours of other non-vocal audio. Finally, the data also includes recordings of 1013 sneezing audio, totalling 13.34 minutes of audio. Hence sneezing is under-represented as a class in the pre-training data. Since such an imbalance in training data can detrimentally affect the performance especially of neural networks [30,31], we have applied the synthetic minority over-sampling technique (SMOTE) [32]. SMOTE oversamples the minor class by creating additional synthetic samples rather than, for example, random oversampling. We have in the past successfully applied SMOTE to address training set class imbalances in cough detection [23] and cough classification [20] based on audio recordings.
In total, therefore, a dataset containing 10.29 hours of audio recordings annotated with four class labels was available to pretrain the neural architectures. The composition of this dataset is summarised in Table 1. All recordings used for pre-training were downsampled at 16 kHz.

Datasets with COVID-19 labels for classification
Three datasets of coughing audio with COVID-19 labels were available for experimentation.

Coswara dataset
This dataset is specifically developed with the testing of classification algorithms for COVID-19 detection in mind. Data collection is web-based, and participants contribute by using their smartphones to record their coughing, breathing and speech. Audio recordings were collected of both shallow and deep breaths as well as speech uttered at a normal and fast pace. However, since the deep breaths consistently outperformed the shallow breaths in our initial experiments, the latter will not be presented in our experiments. At the time of writing, the data included contributions from participants located on five different continents [20,21,33]. Figure 1 and Figure 2 show examples of Coswara breaths and normal-paced count from one to twenty respectively, collected from both COVID-19 positive and COVID-19 negative subjects. It is evident that breaths have more higher-frequency content than speech and interesting to note that COVID-19 breaths are 30% shorter than non-COVID-19 breaths ( Table 2). All audio recordings were pre-processed to remove periods of silence to within a margin of 50 ms using a simple energy detector.   Figure 1: Pre-processed breath signals from both COVID-19 positive and COVID-19 negative subjects in the Coswara dataset. Breaths corresponding to inhalation are marked by arrows, and are followed by an exhalation.  ComParE dataset contains recordings, of both coughs and speech, where the latter is the utterance 'I hope my data can help to manage the virus pandemic' in the speaker's language of choice.

Sarcos dataset
This dataset was collected in South Africa as part of this research and currently contains recordings of coughing by 18 COVID-19 positive and 26 COVID-19 negative subjects. Audio was pre-processed in the same way as the Coswara data. Since this dataset is very small, we have used it in our previous work exclusively for an independent validation [20]. In this study, however, it has also been used to fine-tune and evaluate pre-trained DNN classifiers by means of transfer learning and the extraction of bottleneck features.

Summary of data used for classification
A summary of the above three datasets is presented in Table 2. We see that the COVID-19 positive class is under-represented in all cases. To address this, we again apply SMOTE during training. We also note that the Coswara dataset contains the largest number of subjects, followed by ComParE and then Sarcos and all recordings were downsampled at 16 kHz.

Primary Feature Extraction
From the time-domain audio signals, we have extracted melfrequency cepstral coefficients (MFCCs) and linearly-spaced log filterbank energies, along with their respective velocity and acceleration coefficients. We have also extracted the signal zero-crossing rate (ZCR) and kurtosis [34], which are indicative respectively of time-domain signal variability and tailedness i.e. the prevalence of higher amplitudes. MFCCs have been very effective mostly in speech [35], but also in discriminating dry and wet coughs [36], and recently in characterising COVID-19 audio [37]. Linearly-spaced log filterbank energies have proved useful in biomedical applications, including cough audio classification [24,25,38]. The ZCR is the number of times the time-domain signal changes sign within a frame, and is an indicator of variability [34].
Features are extracted from overlapping frames, where the frame overlap δ is computed to ensure that the audio signal is always divided into exactly S frames, as illustrated in Figure 3. This ensures that the entire audio event is always captured within a fixed number of frames, which allows a fixed input dimension to be maintained while preserving the general overall temporal structure of the sound. Such fixed two-dimensional feature dimensions are particularly useful for the training of DNN classifiers, and has been found to perform well in previous experiments [20].  The input feature matrix to the classifiers has the dimension of (3M+2, S) for M MFCCs along with their M velocity and M acceleration coefficients, as shown in Figure 3. Similarly, for linearly spaced filters, the dimension of the feature matrix is (3B + 2, S).
We will refer to the features described in this section as primary features (PF) to distinguish them from the bottleneck features (BNF), described in Section 5.

Transfer Learning architecture
Since the audio datasets with COVID-19 labels described in Section 2.2 are small, they may lead to overfitting when training deep architectures. Nevertheless, in previous work we have found that deep architectures perform better than shallow classifiers when using these as training sets [20]. In this work, we consider whether the classification performance of such DNNs can be improved by applying transfer learning.
To achieve this, we use the datasets containing 10.29 hours of audio, labelled with four classes: cough, sneeze, speech and noise, but do not include COVID-19 labels ( Table 1 in Section 2.1). This data is used to pre-train three deep neural architectures: a CNN, an LSTM and a Resnet50. The feature extraction hyperparameters: M = 39, F = 2 10 and S = 150 delivered good performance in our previous work [20] and thus have also been used here ( Table 4).
The CNN consists of three convolutional layers, with 256, 128 and 64 (2×2) kernels respectively and each followed by (2,2) max-pooling. The LSTM consists of three layers with 512, 256 and 128 LSTM units respectively, each including dropout with a rate of 0.2. A standard Resnet50, as described in Table 1 of [39], has been implemented with 512-dimensional dense layers.
During pre-training, all three networks (CNN, LSTM and Resnet50) are terminated by three dense layers with dimensionalities 512, 64 and finally 4 to correspond to the four classes mentioned in Table 1. Relu activation functions were used throughout, except in the four-dimensional output layer which was softmax. All the above architectural hyperparameters were chosen by optimising the four-class classifiers during crossvalidation (Table 4).
After pre-training on the datasets described in Section 2.1, the 64 and 4-dimensional dense layers terminating the network were discarded from the CNN, the LSTM and the Resnet50. This left three trained deep neural networks, each accepting the same input dimensions and each with a 512-dimensional relu output layer. The parameters of these three pre-trained networks were then fixed for the remaining experiments.
In order to obtain COVID-19 classifiers by transfer learning, two dense layers are added after the 512-dimensional output layer of each of the three pre-trained deep networks. The final layer is a two-dimensional softmax, to indicate COVID-19 positive and negative classes respectively. The dimensionality of the penultimate layer was also considered to be a hyperparameter and was optimised during nested k-fold crossvalidation. Its optimal value was found to be 32 for all three architectures. The transfer learning process for a CNN architecture is illustrated in Figure 4.

Bottleneck Features
The 512-dimensional output of the three pre-trained networks described in the previous section has a much lower dimensionality than the (3M + 2, S) i.e. (3 × 39 + 2) × 150 = 17850 dimensional input matrix consisting of primary features (Table 4). Therefore, the output of this layer can be viewed as a bottleneck feature vector [40][41][42]. In addition to fine-tuning, where we Table 4: Hyperparameters of the pre-trained networks: Feature extraction hyperparameters were adopted from the optimal values in previous related work [20], while classifier hyperparameters were optimised on the pre-training data using crossvalidation. FEATURE  add terminating dense layers to the three pre-trained networks and optimise these for the binary COVID-19 detection task as shown in Figure 4, we have trained logistic regression (LR), support vector machine (SVM), k-nearest neighbour (KNN) and multilayer perceptron (MLP) classifiers using these bottleneck features as inputs. Bottleneck features computed by the CNN, the LSTM or the Resnet50 were chosen based on the one which performed better in the corresponding transfer learning experiments. Since the Resnet50 achieved higher development set AUCs than the CNN and the LSTM during transfer learning, it was used to extract bottleneck features on which the LR, SVM, KNN and MLP classifiers were trained.

Experimental Method
We have evaluated the effectiveness of transfer learning (Section 4) and bottleneck feature extraction (Section 5) using CNN, LSTM and Resnet50 architectures in improving the performance of COVID-19 classification based on cough, breath and speech audio signals. In order to place these results in context, we provide two baselines.
1. As a first baseline, we train the three deep architectures (CNN, LSTM and Resnet50) directly on the primary features extracted from data containing COVID-19 labels (as described in Section 2.2) and hence skip the pretraining. Some of these baseline results were developed in our previous work [20].
2. As a second baseline, we train shallow classifiers (LR, SVM, KNN and MLP) on the primary input features (as described in Section 3), also extracted from the data con-  Figure 4: Transfer Learning Architecture for a CNN. Cross-validation on the pre-training data determined the optimal CNN architecture to have three convolutional layers with 256, 128 and 64 (2×2) kernels respectively and each followed by (2,2) max-pooling. The convolutional layers were followed by two dense layers with 512 and 64 relu units each, and the network was terminated by a 4-dimensional softmax. To apply transfer learning, the final two layers were removed and replaced with a new dense layer and a terminating 2-dimensional softmax to account for COVID-19 positive and negative classes. Only this newly added portion of the network was trained for classification on the data with COVID-19 labels. In addition, the outputs of the third-last layer (512-dimensional dense relu) from the pre-trained network were used as bottleneck features.
The performance of these baseline systems will be compared against:

Hyperparameter Optimisation
Hyperparameters for three pre-trained networks have already been described in Section 4 and are listed in Table 4. The remaining hyperparameters are those of the baseline deep classifiers (CNN, LSTM and Resnet50 without pre-training), the four shallow classifiers (LR, SVM, KNN and MLP), and the dimensionality of the penultimate layer for the deep architectures during transfer learning.
With the exception of Resnet50, all these hyperparameters optimisation and performance evaluation has been performed within the inner loops of a leave-p-out nested cross-validation scheme [43]. Due to the excessive computational requirements of optimising Resnet50 metaparameters within the same crossvalidation framework, we have used the standard 50 skip layers in all experiments [39]. Classifier hyperparameters and the values considered during optimisation are listed in Table 5. A five-fold split, similar to that employed in [20], was used for the nested leave-p-out cross-validation.

Classifier Evaluation
Receiver operating characteristic (ROC) curves were calculated within both the inner and outer loops of the leave-p-out crossvalidation scheme described in the previous section. The innerloop ROC values were used for the hyperparameter optimisation, while the average of the outer-loop ROC values indicate fi-nal classifier performance on the independent held-out test sets. The AUC score indicates how well the classifier has performed over a range of decision thresholds [44]. The threshold that achieves an equal error rate (γEE) was computed from these curves.
We note the mean per-frame probability that an event such as a cough is from a COVID-19 positive subject byP : where K indicates the number of frames in the cough and P (Y = 1|Xi, θ) is the output of the classifier for feature vector Xi and parameters θ for the i th frame. Now we define the indicator variable C as: We then define two COVID-19 index scores as CI1 and CI2, N1 as the number of coughs from the subject in the recording and N2 as the total number of frames of cough audio gathered from the subject in Equations 3 and 4 respectively.
Hence Equation 1 computes a per-cough average probability while Equation 4 computes a per-frame average probability. For the shallow classifiers, the use of one of Equations 3 and 4 was considered an additional hyperparameter during cross-validation, and it was found that taking the maximum value of the index scores consistently led to the best performance.
The average specificity, sensitivity and accuracy, as well as the AUC together with its standard deviation (σAUC ) are shown Table 5: Classifier hyperparameters, optimised using leave-pout nested cross-validation.

Hyperparameters
Classifier Range  Tables 6, 7 and 8 for cough, breath and speech respectively. These values have all been calculated over the outer folds during nested cross-validation. Hyperparameters producing the highest AUC over the inner loops have been noted as the 'best classifier hyperparameter'.

Experimental Results
COVID-19 classification performance based on cough, breath and speech are presented in Tables 6, 7 and 8 respectively. These tables include the performance of baseline deep classifiers without pre-training, deep classifiers trained by transfer learning (TL), shallow classifiers using bottleneck features (BNF) and baseline shallow classifiers trained directly on the primary features (PF). The best performing classifiers appear first for each dataset and the baseline results are shown towards the end. Each system is identified by an 'ID'.

Coughs
We have found in our previous work [20] that, when training a Resnet50 on only the Coswara dataset, an AUC of 0.976 (σAUC = 0.018) can be achieved for the binary classification problem of distinguishing COVID-19 coughs from healthy coughs. These results are reproduced as baseline systems C8, C9 and C10 in Table 6. The improved results achieved by transfer learning are indicated by systems C1 to C7 in the same table.
Specifically, system C1 shows that, by applying transfer learning as described in Section 4, the same Resnet50 architecture can achieve an AUC of 0.982 (σAUC = 0.002). The entries for systems C2 and C3 show that pre-training also improves the AUCs achieved by the deep CNN and LSTM classifiers from 0.953 (system C9) to 0.972 (system C2) and from 0.942 (system C10) to 0.964 (system C3) respectively. Of particular note in all these cases is the substantial decrease in the standard deviation of the AUC (σAUC ), observed during cross-validation when implementing transfer learning. This indicates that pretraining leads to classifiers with more consistent performance on the unseen test data. The Sarcos dataset is much smaller than the Coswara dataset and too small to train a deep classifier directly. For this reason, it was used only as an independent validation dataset for classifiers trained on the Coswara data in our previous work [20]. It can however be used to fine-tune pre-trained classifiers during transfer learning, and the resulting performance is reflected by systems C11 to C17 in Table 6. Previously an AUC of 0.938 (system C20) was achieved when using Sarcos as an independent validation data and applying sequential forward selection (SFS). Here, we find that transfer learning applied to the Resnet50 model results in an AUC of 0.961 (system C11) and a lower standard deviation (σAUC =0.003) than that observed for the Coswara dataset [20]. As an additional experiment, we apply the Resnet50 classifier trained by transfer learning using the Coswara data to the Sarcos data, thus again using the latter as an independent validation set. The resulting performance is indicated by system C18, while the previous baselines are repeated as systems C19 and C20. System C18 achieves an AUC of 0.954, which is only slightly below the 0.961, achieved by system C11 where the pre-trained model used the Sarcos data for fine-tuning, and slightly higher than the AUC of 0.938 achieved by system C20 which is the baseline LSTM trained on Coswara without transfer learning but employing SFS [45]. This supports our earlier observation that transfer learning appears to lead to more robust classifiers that can generalise to other datasets. Due to the extreme computational load, we have not yet been able to evaluate SFS within the transfer learning framework.
For the ComParE dataset, we have included shallow classifiers trained directly on the primary input features (KNN+PF, MLP+PF, SVM+PF and LR+PF). These are the baseline systems C29 to C32 in Table 6. The best-performing shallow classifier is C29, where a KNN used 60 linearly spaced filterbank log energies as features. System C28 is the result of applying SFS to system C29. In this case, SFS identifies the top 12 features based on the development sets used during nested cross-validation, and results in the best-performing shallow system with an AUC of 0.944. This represents a substantial improvement over the AUC of 0.855 achieved by the same system without SFS (system C29). Systems C21 to C27 in Table 6 are obtained by transfer learning using the ComParE dataset. These show improved performance over the shallow classifiers without SFS. In particular, after transfer learning, the Resnet50 achieves almost the same AUC as the best ComParE system (system C28) with a lower σAUC .
When considering the performance of the shallow classifiers trained on the bottleneck features across all three datasets in Table 6, we see that a consistent improvement over the use of primary features with the same classifiers is observed. The ROC curves for the best-performing COVID-19 cough classifiers are shown in Figure 5. Table 7 demonstrates that COVID-19 classification is also possible on the basis of breath signals. The baseline systems B8, B9 and B10 are trained directly on the primary features, without pre-training. By comparing these baselines with B1, B2 and B3, we see that transfer learning leads to a small improvement Table 6: COVID-19 cough classifier performance: For the Coswara, Sarcos and ComParE datasets the highest AUCs were 0.982, 0.961 and 0.944 respectively and achieved by a Resnet50 trained by transfer learning in the first two cases and a KNN classifier using 12 primary features determined by sequential forward selection (SFS) in the third case. When Sarcos as a validation-only datset for a classifier trained on the Coswara data, an AUC of 0.954 is achieved.
in Figure 6. As it was observed for coughs, the standard deviation of the AUC (σAUC ) is consistently lower for the pre-trained networks.

Speech
Although not as informative as cough or breath audio, COVID-19 classification can also be achieved on the basis of speech audio recordings. For Coswara, the best classification performance (AUC = 0.893) was achieved by a Resnet50 after applying transfer learning (system S1). For the ComParE data, the top performer (AUC = 0.923) was an SVM trained on the bottleneck features (system S24). However, the Resnet50 trained by transfer learning performed almost equally well, with an AUC of 0.914 (system S21). Furthermore, while good performance was also achieved when using the deep architectures without applying the transfer learning process (systems S8-S10, S18-S20 and S28-S32), this again was at the cost of a substantially higher standard deviation σAUC . Finally, for the Coswara data, performance was generally better when speech was uttered at a normal pace rather than a fast pace. The ROC curves for the best-performing COVID

Discussion
Previous studies have shown that it is possible to distinguish between the coughing sounds made by COVID-19 positive and COVID-19 negative subjects by means of automatic classification and machine learning. However, the fairly small size of datasets with COVID-19 labels limits the effectiveness of these techniques. The results of the experiments we have presented in this study show that larger datasets of other vocal and respiratory audio that do not include COVID-19 labels can be leveraged to improve classification performance by applying transfer learning. Specifically, we have shown that the accuracy of COVID-19 classification based on coughs can be improved by transfer learning for two datasets (Coswara and Sarcos) while almost optimal performance is achieved on a third dataset (ComParE). A similar trend is seen when performing COVID-19 classification based on breath and speech audio. However, these two types of audio appear to contain less distinguishing information, since the achieved classification performance is a little lower than it is for cough. Our best cough classification system has an area under the ROC curve (AUC) of 0.982, despite being trained on what remains a fairly small COVID-19 dataset with 1171 participants (92 COVID-19 positive and 1079 negative). Other research reports a similar AUC but using a much larger dataset with 8380 participants (2339 positive and 6041 negative) [46]. Application of transfer learning on all three audio types presented in this study shows better performance than recent studies as well [47]. While our experiments also show that shallow classifiers, when used in conjunction with feature selection, can in some cases match or surpass the performance of the deeper architectures; a pre-trained Resnet50 architecture provides consistent optimal or near-optimal performance across all three types of audio signals and datasets. Due to the very high computational cost involved, we have not yet applied such feature selection to the deep architectures themselves, and this remains part of our ongoing work. Another important observation that we can make for all three types of audio signals is that transfer learning strongly reduces the variance in the AUC (σAUC ) exhibited by the deep classifiers during cross-validation (Table 6, 7 and 8). This suggests that transfer learning leads to more consistent classifiers that are less prone to over-fitting and better able to generalise on the unseen data. This is important because robustness to variable testing conditions is essential in implementing COVID-19 classification as a method of screening.
An informal listening assessment of the Coswara and the ComParE data indicates that the former has greater variance and more noise than the latter. Our experimental results presented in Table 6, 7 and 8 found that, for speech classification on a noisy data, fine-tuning the pre-trained networks demonstrates better performance, while for cleaner data, extracting bottleneck features and then applying a shallow classifier exhibits better performance. It is interesting to note that MFCCs are always the features of choice for this noisier dataset, while the log energies of linear filters are often preferred for the less noisy data. Although all other classifiers have shown the best performance when using these log-filterbank energy features, MLP has always performed the best on MFCCs and has been proved to be the best classifier in classifying COVID-19 speech. A similar conclusion was also drawn in [24], where coughs were recorded in a controlled environment with little environmental noise. A higher number of segments also generally leads to better performance as it allows the classifier to find more detailed temporal patterns in the audio signal.

Conclusions
In this study, we have demonstrated that transfer learning can be used to improve the performance and robustness of the DNN classifiers for COVID-19 detection in vocal audio such as cough, breath and speech. We have used a 10.29 hour audio data corpus, which do not have any COVID-19 labels, to pre-train a CNN, an LSTM and a Resnet50. This data contains four classes: cough, sneeze, speech and noise. In addition, we have used the same architectures to extract bottleneck features by removing the final layers from the pre-trained models. Three smaller datasets containing cough, breath and speech audio with COVID-19 labels were then used to fine-tune the pretrained COVID-19 audio classifiers using nested leave-p-out cross-validation. Our results show that a pre-trained Resnet50 classifier that is either fine-tuned or used as a bottleneck extractor delivers optimal or near-optimal performance across all datasets and all three audio classes. The results show that trans-fer learning using the larger dataset without COVID-19 labels led not only to improved performance, but also to a smaller standard deviation of the classifier AUC, indicating better generalisation. The use of bottleneck features, which are extracted by the pre-trained deep models and therefore also a way of incorporating out-of-domain data, also provided a reduction in this standard deviation and near-optimal performance. The experiments show that cough audio carries the strongest COVID-19 signatures, followed by breath and speech. The best-performing COVID-19 classifier achieved an area under the ROC curve (AUC) of 0.982 for cough, followed by an AUC of 0.942 for breath and 0.923 for speech.
Finally, we note that, for the shallow classifiers, hyperparameter optimisation selected a higher number of MFCCs and also a more densely populated filterbank than what is required to match the resolution of the human auditory system. This agrees with an observation we have already made in our previous work that the information used by the classifiers to detect COVID-19 signature is at least to some extent not perceivable by the human ear.
We conclude that successful classification is possible for all three classes of audio considered. However, deep transfer learning improves COVID-19 detection on the basis of cough, breath and speech signals, yielding automatic classifiers with higher accuracies and greater robustness. This is significant since such COVID-19 screening is inexpensive, easily deployable, non-contact and does not require medical expertise or laboratory facilities. Therefore it has the potential to decrease the load on the health care systems.
As a part of ongoing work, we are considering the application of feature selection in the deep architectures, the fusion of classifiers using various audio classes like cough, breath and speech, as well as the optimisation and adaptation necessary to allow deployment on a smartphone or similar mobile platform.