{'fraction_non_alphanumeric': 0.03238062833163208, 'fraction_numerical': 0.012135647045480322, 'mean_word_length': 4.913816230717639, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improving Polyphonic Sound Event Detection on Multichannel Recordings with the S{\o}rensen-Dice Coefficient Loss and Transfer Learning

The S{\o}rensen--Dice Coefficient has recently seen rising popularity as a loss function (also known as Dice loss) due to its robustness in tasks where the number of negative samples significantly exceeds that of positive samples, such as semantic segmentation, natural language processing, and sound event detection. Conventional training of polyphonic sound event detection systems with binary cross-entropy loss often results in suboptimal detection performance as the training is often overwhelmed by updates from negative samples. In this paper, we investigated the effect of the Dice loss, intra- and inter-modal transfer learning, data augmentation, and recording formats, on the performance of polyphonic sound event detection systems with multichannel inputs. Our analysis showed that polyphonic sound event detection systems trained with Dice loss consistently outperformed those trained with cross-entropy loss across different training settings and recording formats in terms of F1 score and error rate. We achieved further performance gains via the use of transfer learning and an appropriate combination of different data augmentation techniques.


INTRODUCTION
Sound event detection (SED) is the task of jointly detecting the onset, offset, and class of a sound event. Polyphonic SED refers specifically to the task of such detection for multiple, potentially overlapping sound events simultaneously. In the past decade, there have been significant advances in the applications of deep learning for SED [1]. The state-of-the-art SED models have been built from convolutional neural networks (CNN) [2], convolutional recurrent neural networks (CRNN) [3], and more recently, Conformer [4].
Majority of works in SED targeted single-channel inputs, due to the availability of large-scale datasets such as AudioSet [5] and FSD50k [6], and ease of practical deployment. For multichannel SED (MSED), there are only a few publicly-available labelled datasets, often small-scale, e.g., TUT Sound Events 2016 [7], and TAU-NIGENS Spatial Sound Events 2021 [8]. In addition, many MSED works naturally focused on multichannel and spatial feature engineering [9,10] due to the different multichannel formats available. As a result, there is a severe lack of literature concerning the impact of different data augmentation techniques, network architectures, and loss functions on the performance of MSED models that are trained on small datasets. In this paper, we present a thorough investigation on the factors affecting the performance of deep polyphonic MSED systems. We demonstrated the effectiveness of using pretrained weights from single-channel model for MSED to tackle the lack of training data. We also investigated the use of dice loss, which has been shown to improve single-channel SED performance [11]. Several experiments on the data augmentation techniques, training chunk durations, pretraining modalities, and multichannel audio formats were performed and their results were analyzed and discussed to better inform our understanding of polyphonic MSED systems. The rest of the paper is organized as follows. Section 2 describes our experiment setups. Section 3 presents the experimental results and discussion. Finally, we conclude the paper in Section 4.

Dataset
In this paper, we used the TAU-NIGENS Spatial Sound Events 2021 (SSE21) dataset [8], which provides two four-channel array formats: first-order ambisonics (FOA) and microphone array (MIC). Each format has 400, 100, and 100 60-second recordings for training, validation, and testing, respectively. Both formats were used to train and evaluate the performance of the SED networks. Since SSE21 is a sound event localization and detection (SELD) dataset with directional interferences, we only take the SED labels. Hence, we have strongly-labeled sound events from the L = 12 target classes and unlabelled out-of-class interferences.

Input features
Polyphonic C-channel audio can generally be modelled in the timefrequency (TF) domain by where Si[t, f ] ∈ C is the i th sound source, Hi[t, f ] ∈ C C is the corresponding array response vector from the source to the sensor array, and N[t, f ] ∈ C C is the noise vector. This representation is typically achieved by applying the short-time Fourier transform (STFT) on the multichannel time-domain audio data. A Hann window of size R = 1024 with hop size 300 was used for this paper. For the FOA format, the array response vector is given by where φi[t] and θi[t] are the time-dependent azimuth and elevation angles of the i th sound source with respect to the array, respectively.
For the microphone format, SSE21 provides a tetrahedral array of microphones mounted on a spherical baffle, whose response has a very complex analytical expression [12, eq. (5)]. From our experience, we have found that a generic far-field array response can usually approximates the theoretical response satisfactorily. The far-field response for the c th channel at time t is given by up to some channel-dependent scaling, where v ≈ 343 m s −1 is the speed of sound, and di,c[t] is the projected difference in distances, in metres, travelled by the i th sound source arriving the c th microphone relative to the reference microphone.
In this paper, 128-bin log-mel spectrograms were used as input features, with cutoff frequencies at 50 Hz and 12 kHz. To facilitate learning, z-score normalization was performed on each channel of the spectrograms along the mel-frequency bin axis.

Data augmentation
As with many audio datasets, SSE21 is relatively small, thus requiring data augmentation. In this paper, we experiment with combinations of four data augmentation techniques: mixup with soft label [13], frequency shifting, channel swapping [14], and a composite technique of cutout [15] and SpecAugment [16].

Loss function
In SED, the number of negative samples often significantly exceeds that of positive samples. Hence, training polyphonic SED systems with binary cross-entropy (BCE) loss often results in suboptimal F1 score. An alternative loss function based on the Sørensen-Dice Coefficient (SDC) [17,18], called Dice loss, was proposed in [19]. Variations of the Dice loss have since become popular as loss functions in semantic segmentation [20] and natural language processing. A variant was also recently used in single-channel SED [11].
The SDC between two sets A and B is defined by where |·| is the set cardinality operator. The SDC is used to gauge the similarity between two sets. In binary classification, the SDC is identical to the F1 score. Generalizing the SDC to tensors with elements in the unit interval, the cardinality operator can be replaced by the L1-norm, · 1, while the intersection operator can be replaced by the Hadamard multiplication, •. As such, a differentiable Dice loss for a batch of N predictions is given by where Yn ∈ [0, 1] T ×L is the prediction tensor, Yn ∈ {0, 1} T ×L is the target tensor, ǫ > 0 is a small stabilizing constant, T is the number of label segments, and L is the number of target sound classes.
In this paper, we experiment with the BCE-Dice loss, which is a combination of the BCE and Dice losses, given by

Network architecture
The general architecture of the SED network used in this paper is that of a CRNN, a de facto standard architecture for SED [1,3]. The input is passed through a CNN backbone and pooled in the frequency axis before being passed into an RNN and a fully-connected (FC) output layer.
In this paper, we experiment with a number of different CNN backbones from both audio and vision domains, such as Pretrained Audio Neural Networks (PANNs) [2], ResNet [21], Effi-cientNet [22], DenseNet [23]. For ease of comparison, we only use a single-layer 256-unit bidirectional gated recurrent unit (BiGRU) as the RNN network. PyTorch Image Models implementations [24] were adapted for all CNN backbones except PANNs.

Transfer learning
Early research works into the applications of deep learning in the audio domain suffered severely from the lack of data. Many early audio systems relied on the AudioSet-pretrained VGGish model [5,25]. The use of intra-modal (audio-for-audio) transfer learning remains popular as seen by the frequent use of PANNs [2] in many audio systems today. Recently, more works have successfully attempted inter-modal (vision-for-audio) transfer learning by fine-tuning networks with CNN backbones pretrained on Ima-geNet [26][27][28][29]. Unfortunately, research into the effect of modalities on transfer learning for audio systems remains very limited [30].
In this paper, we investigate the effect of modalities in transfer learning for SED, using CNN backbones pretrained on either single-channel AudioSet [5] or three-channel ImageNet [29]. Since SSE21 provides four-channel inputs, the weight replication technique from [24] is used on the first convolutional layer in order to match the original pretrained weights to the four-channel inputs.

Training procedures
Recordings from the training set are chunked into four-second segments with a hop size of 0.5 s unless otherwise stated. All models are trained for 30 epochs with a batch size of 32. We use the Adam optimizer [31] with a learning rate schedule shown in Figure 1.

Evaluation metrics
We evaluate the SED performance using the segment-based F1 score and error rate (ER) with a segment length of 1 s, which have become the standard metrics for SED [1,32]. We combine the ER and F1 score into a single metric for checkpoint selection as follows: SEDE = 0.5 · ER + 0.5 · (1 − F1 ). A lower SEDE generally indicates better model performance. All reported results were evaluated on the test set with a binarization threshold of 0.3. The composite cutout-SpecAugment technique (CO) goes as follows. First, one of the these three techniques is chosen at random: single cutout, multiple cutouts, and SpecAugment. Single cutout fills a same-ratio rectangular section randomly located on the spectrogram, sized between 2 % and 30 % of the total area, with a constant value drawn randomly from the support of the original input. Multiple cutouts is similar but with 8 randomly located sections of size 8 bins by 8 frames each. SpecAugment uses a time stripe with a width up to 15 % of the frames and a frequency strip with a width up to 20 % of the mel bins.
The results are shown in Table 1. For the FOA format, the ER is the lowest with all four augmentation techniques applied, while the F1 score is the highest when all except mixup was applied, although the F1 score for the all-four setting is very close to the no-mixup setting. For the MIC format, the best performance is achieved using the combination of all techniques except mixup. In fact, adding mixup seems to generally worsen the performance compared to the same combination without mixup. FS seems particular useful for both format. CS is useful for FOA format while CO is useful for MIC format. Majority of methods work better in combination than standalone. Interestingly, with or without data augmentation, MIC format provides better performance than FOA format.
We suspect that the difference in results is due to the difference in how spatial information is stored in the FOA and MIC formats. All channels of MIC formats tend to have similar magnitude information since spatial information is mostly encoded in the phase. The drop in performance when channel swapping was applied on MIC format supports this; we suspect swapping in the MIC format introduces little to no new information in the augmented samples. Moreover, sound event overlaps on spectrograms in the MIC format are often irresolvable without phase information. Hence, mixup likely made the overlapping sound events too difficult to detect by the network. On the other hand, the FOA format encodes spatial information via the relative intensity across the channels with little spatial information in the phase. As such, when mixup is applied, sound events originating from distinct directions are more likely to

Experiment 2: Loss function
The second experiment investigates the effect of loss functions on the performance of SED models with and without transfer learning. The experiment settings were identical to that of Experiment 1.
The results for this experiment are shown in Table 2. For both formats, the best performance was achieved with both transfer learning and the BCE-Dice loss function. This result is somewhat expected as the Dice loss is generally more robust to positivenegative imbalance in the dataset than the BCE loss, and can be considered a soft approximation of the binary F1 score [33]. Interestingly, using only either the BCE-Dice loss or pretraining worsened the performance in the MIC format, but using both improved the performance.This is not observed in the FOA format, where using either technique improved the performance, and using both improved the performance further.

Experiment 3: Training chunk size
The third experiment investigates the impact of the training chunk size on the SED performance. For reference, event lengths for SSE21 have a median of 3.2 s and a mean of 8.3 s. We used both   Table 4: SED performance w.r.t. number of channels.
transfer learning and the BCE-Dice loss for training. The rest of the settings followed Experiment 2.
The results for this experiment are shown in Table 3. For the FOA format, increasing the chunk size from 4 s to 8 s barely improved the performance, but increasing to 12 s resulted in a some appreciable gain. The results for the MIC format are less consistent, with the 8 s chunk performing the worst. Further analysis of SSE21 revealed some sound classes with a significant number of events whose durations are beyond 8 s. Insufficient coverage by the 8 s chunk may explain the lack in performance gain, as training difficulty increased beyond the additional information gained.

Experiment 4: Input channels
The fourth experiment compares the performance of SED networks using single-channel inputs against those using the full four-channel array inputs. For the single-channel setting, the omnidirectional (W) channel was used for the FOA format while the first channel was used for the MIC format. A chunk size of 4 s was used and the rest of the settings followed Experiment 3.
The results are shown in Table 4. With the addition of more channels, the FOA format achieved very little performance gain, if any, whereas the MIC format gained significant improvement. The similar performances for the FOA format are likely because the information in the W channel was already acquired indirectly via multiple microphones in an ambisonic array (32 sensors for SSE21). On the other hands, each channel in the MIC format only acquired information from a single sensor (which is also more representative of a single-sensor signal acquisition in practice). As such, the information gained from adding more channels in the MIC format is more significant, compared to the mostly spatial information gained from more channels in the FOA format.

Experiment 5: Modalities in transfer learning
In the last experiment, we investigate the impact of pretraining modalities in transfer learning (TL) on the MSED performance with various CNN backbones. The settings followed Experiment 4 with all channels as inputs.
The results for this experiment are shown in Table 5. For both formats, the pretrained networks generally perform better than their counterparts trained from scratch. This confirms the conventional  In terms of architecture, it is clear that models with ResNetbased or CNN14 backbones outperformed those with EfficientNet or DenseNet backbone when trained from scratch. DenseNet architecture is a known to cause aliasing problems [34] which may explain the poor performance. EfficientNet has building blocks which consist of information compressing layers, potentially causing in compounding information loss when trained on small datasets.
In terms of modalities, comparing the PANNs pretrained on AudioSet to other CNNs pretrained on ImageNet, it is clear networks with AudioSet-pretrained backbones outperformed those with ImageNet-pretrained backbones. This is consistent with the findings from [30] where intra-modal transfer learning outperformed inter-modal ones. Since higher-level features in audio spectrograms are very different from higher-level features in images, we suspect that higher-level representation prelearning of audio-related patterns may only be achieved via intra-modal pretraining, even if pretraining on vision datasets gives the network a headstart in learning lower-level visual representation.

CONCLUSION
In this paper, we performed a thorough investigation on the factors affecting the performance of deep multichannel polyphonic SED systems: data augmentation techniques, loss functions, training chunk durations, pretraining modalities, and multichannel audio formats. A proper combination of data augmentation techniques can mitigate the problem of small datasets, and significantly improve the model performance. Further improvement can be achieved by using the BCE-Dice loss and transfer learning. We showed that both inter-and intra-model transfer learning from other datasets increases the SED performance on multi-channel datasets, even with channel mismatch, with intra-model transfer learning providing higher performance gains. Interestingly, across many settings, the best performances of MIC format are slightly higher than those of FOA format.