{'fraction_non_alphanumeric': 0.03686511940800538, 'fraction_numerical': 0.012243525058863102, 'mean_word_length': 4.834379905808477, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Joint Analysis of Acoustic Event and Scene Based on Multitask Learning

Acoustic event detection and scene classification are major research tasks in environmental sound analysis, and many methods based on neural networks have been proposed. Conventional methods have addressed these tasks separately; however, acoustic events and scenes are closely related to each other. For example, in the acoustic scene `office', the acoustic events `mouse clicking' and `keyboard typing' are likely to occur. In this paper, we propose multitask learning for joint analysis of acoustic events and scenes, which shares the parts of the networks holding information on acoustic events and scenes in common. By integrating the two networks, we expect that information on acoustic scenes will improve the performance of acoustic event detection. Experimental results obtained using TUT Sound Events 2016/2017 and TUT Acoustic Scenes 2016 datasets indicate that the proposed method improves the performance of acoustic event detection by 10.66 percentage points in terms of the F-score, compared with a conventional method based on a convolutional recurrent neural network.


INTRODUCTION
There has been increasing interest in analyzing various sounds in real-life environments such as cooking sounds in a kitchen or car passing sounds in a street [1]. Various applications can be expected from the automatic analysis of environmental sounds, for example, abnormal sound detection systems [2], automatic life-logging systems [3,4], surveillance systems [5,6], and hearing-impaired support systems [7,8]. In environmental sound analysis, the following two tasks have mainly been studied: acoustic event detection (AED) and acoustic scene classification (ASC). AED is the task of detecting acoustic event labels and their onset/offset in an audio recording, where an acoustic event indicates a type of sound such as "mouse clicking," "people talking," or "bird singing." ASC is the task of predicting acoustic scene labels from a relatively long duration recording, where an acoustic scene indicates a recording situation with human activity such as "office," "train," or "cooking." For AED and ASC, many methods based on the Gaussian mixture model (GMM) [9,10], hidden Markov model (HMM) [11,12], and support vector machine (SVM) [13], have been proposed. As alternative approaches for AED, those based on non-negative matrix factorization (NMF) [14,15] have been studied. Such approaches enable the analysis of polyphonic acoustic events by decomposing their spectrum into product of a basis and activation matrix, where each basis and each activation vector indicate a single acoustic event and the active duration of the corresponding event, respectively. More recently, developments in acoustic event and scene analysis have led to renewed interest in deep neural networks (DNNs) [16,17]. The DNN-based approaches achieve better results than the conventional methods based on HMM, GMM, and SVM.
In AED and ASC, most of the studies have addressed acoustic events and scenes analysis separately; in many cases however, acoustic events and scenes are related to each other. For example, in the acoustic scene "office", the acoustic events "mouse clicking" and "keyboard typing" are likely to occur, whereas the acoustic events "car" and "bird singing" do not tend to occur. In other words, when analyzing the acoustic events "mouse clicking" and "keyboard typing," it is expected that information on the acoustic scene "office" will help in detecting these acoustic events. On the basis of this idea, AED utilizing information of the acoustic scene in an unsupervised manner [18,19] and ASC taking information of acoustic events into account [20], which are based on Bayesian generative models, have been proposed. However, these conventional methods do not estimate both acoustic events and scenes explicitly. Moreover, these methods cannot be applied to state-of-the-art neural network-based methods.
In this paper, we present a new method for the joint analysis of acoustic events and scenes based on multitask learning. The contribution of this work is summarized as follows: • we propose a method for joint analysis of acoustic events and scenes using multitask learning combining state-of-the-art AED and ASC approaches.
• we demonstrate that the multitask learning-based method improves the AED performance.
The remainder of this paper is structured as follows. In Sec. 2, conventional methods for AED and ASC are first discussed and the proposed method for joint analysis of acoustic events and scenes based on multitask learning is then introduced. In Sec. 3, the experiments carried out to evaluate the performance of event detection and scene classification are reported. Finally, we summarize and conclude this paper in Sec. 4.

Conventional Method for Event Detection and Scene Classification
In this section, we review conventional AED and ASC methods based on neural networks. AED involves the estimation of acoustic event labels and their onset/offset times, where acoustic events may overlap in the time axis. In recent years, many neural network based methods, such as a convolutional neural network (CNN)based approach [21] and a recurrent neural network (RNN)-based approach [22], have been proposed. Specifically, it has been reported that neural networks combining a CNN and a bidirectional gated recurrent unit (BiGRU) successfully analyze acoustic events with reasonable performance [23,24]. In the CNN-BiGRU network, the acoustic feature X ∈ R D×T , which is a time and frequencyspectrum representation, is input to the network. Here, D and T are the number of frequency bins and the number of time frames of the input acoustic feature, respectively. The convolution layer convolutes the input feature map with two-dimensional filters, then max pooling is conducted to reduce the dimension of the feature map. The output of the convolutional layer V ∈ R D ′ ×T ×C is then concatenated as V ′ = (x1, x2, . . . xt, . . . , xT ) ∈ R (D ′ ·C)×T , and then V ′ is fed to the BiGRU layer, where t and C are respectively the time index and the number of filters of the convolution layer. After that, the output vector ht is calculated using the following equations: where W and U are parameter matrices and b is a bias vector. Superscripts f and b are the forward and backward networks, respectively. Subscripts g and r indicate the update gate and reset gate, respectively. g, r, ⊙, and σ indicate the update gate vector, reset gate vector, Hadamard product, and sigmoid function, respectively. The BiGRU layer is followed by a fully connected layer, which is the output layer of the network calculated as The CNN-BiGRU network is optimized under the following sigmoid cross-entropy objective function E1(Θ1) using the backpropagation through time (BPTT): where m and zm,t are the index of the acoustic event category and the target label in time frame t, which is 1 if acoustic event m is active in time frame t, and 0 otherwise. ASC is the task of estimating the acoustic scene with which a sound clip is most associated. It has been reported that CNNbased methods achieve state-of-the-art performance in ASC [25,26] In CNN-based ASC, the time-frequency representation of acoustic  Figure 1: Example of network structure for proposed multitask learning of acoustic events and scenes feature X is fed to a convolutional layer. The CNN layer is followed by a fully connected layer, which is the output layer of the network calculated as where S indicates the softmax function. This network is optimized under the following softmax cross-entropy objective function E2(Θ2): where n and zn are the number of acoustic scene categories and the target label, respectively.

Proposed Method
The conventional methods for AED and ASC have been studied separately. However, many acoustic events and scenes are related; thus, we consider that the middle layers of the networks for both AED and ASC extract features common to the acoustic event and scene. On the basis of this idea, we propose the multitask learning of acoustic events and scenes, which shares the parts of the networks holding information of acoustic events and scenes in common. The multitask learning enables us to jointly analyze multiple tasks that are related to each other [27]. In speech processing, it has been reported that multitask learning successfully improves the performance of speech recognition [28,29]. Moreover, it has also been reported that when there is a difference in the degree of difficulty between tasks, the performance of the harder task is improved using the information obtained from the easier task [27]. In this work, we consider that AED is the harder task and ASC is the easier task, and expect that the information of acoustic scenes will help AED. The concept of the proposed method is shown in Fig. 1, where we refer to the shared part of the network as "shared layers." In this network, we apply the CRNN as the event detection network and the CNN as the scene classification network, and the CNN layers are shared between the event detection and scene classification where α is the weight of the acoustic scene classification loss. Note that the proposed method can also be applied to different networks from that shown in Fig. 1 when the networks can be optimized by Eq. (12).

Experimental Conditions
We evaluated the performance of AED and ASC using the TUT Sound Events 2016 [30] development, the 2017 [25] development, and the TUT Acoustic Scenes 2016 [30] development datasets. In these datasets, we used sound clips including four acoustic scenes; "home," "residential area" (TUT Sound Events 2016), "city center" (TUT Sound Events 2017), and "office" (TUT Acoustic Scenes 2016), which contain 192 min of audio. Here, the acoustic scene "office" did not have acoustic event labels; thus, we manually annotated event labels to the sound clips with the protocol described in [30] and [25]. These sound clips contain the 25 acoustic events listed in Table 3. We used log-mel energies of 64 dimensions as acoustic features, which were calculated for each 40 ms time frame with 50% overlap. The acoustic features were then input to the network shown in Fig. 1. AED and ASC results were obtained from the outputs of the network, in which active acoustic events were predicted using 0.5 as the threshold of activation. The experiments were conducted using a four-fold cross-validation setup. Other experimental conditions are listed in Table 1.

Metrics
In AED, since acoustic events may overlap, the event detection performance is evaluated on the basis of a segment-based F-score and error rate (ER) [31]. To calculate the segment-based F-score, the precision and recall are first calculated as where TP, FP, and FN are the total counts of true positive, false positive, and false negative for all time frames and acoustic events, respectively. The segment-based F-score is then calculated as To calculate the segment-based ER, substitutions (S), deletions (D), and insertions (I) are first calculated as where k indicates the index of the time frame. The ER is then calculated as where K and N (k) are the total number of time frames and the number of acoustic events in time frame k, respectively. On the other hand, in ASC, acoustic scenes do not overlap; thus, it is regarded as a simple classification task. The F-score is calculated using Eqs. (13) - (15), where TP, FP, and FN are the total counts of true positive, false positive, and false negative for all sound clips, respectively.

Experimental Results
As comparative methods, we evaluated the detection of an event using CRNN (referred to as CRNN (event)) and the classification of a scene using CNN (referred to as CNN (scene)). CRNN (event) had the same structures and parameters as those of the shared and event layers in Fig. 1, whereas CNN (scene) had the same structures and parameters as those of the shared and scene layers.
Experimental results are shown in Table 2. The results show that the proposed multitask-based method enables the joint analysis of acoustic events and scenes with a reasonable performance compared with CRNN (event) and CNN (scene). In particular, when α = 0.01, the proposed method improved the performance by 10.66 percentage points in terms of the F-score of the event detection result compared with that of the conventional method. This result indicates that information of acoustic scenes improves the performance of AED.
Further evaluations were conducted to investigate how the proposed method performs with various settings of α. The performances in event detection and scene classification for various α are shown in Fig. 2. The results show that when α is less than 1.0, the performance of AED is better than that of the conventional CRNN (event). When α is larger than 0.1, the performance of ASC improved, whereas that of AED was worse than that of the conventional CRNN (event). This implies that AED is a harder task than ASC, and thus, the parameter α should be set to a small value to achieve AED with better performance than the conventional methods.
For a more detailed examination, we list the detection results for each event in Table 3. In this experiment, we evaluated the detection performance using α = 0.01. The results show that the pro-posed method improves the F-score and ER in many of the acoustic events. For example, the acoustic events "dishes", "people walking," and "washing dishes" can be detected more accurately by the proposed method; the F-scores of these acoustic events increase by 4.71, 7.29, and 10.75 percentage points, respectively, compared with the conventional CRNN (event). On the other hand, the detection performance for "brakes squeaking" and "car" do not improve. This is because they may occur in both the acoustic scenes "residential area" and "city center," and thus, information of acoustic scenes may not effectively improve the performance of AED.

CONCLUSION
In this paper, we proposed multitask learning for the joint analysis of acoustic events and scenes. In the proposed method, we applied a CRNN as the event detection network and a CNN as the scene classification network, and the CNN layers in both network were shared. Then, we integrated their objective functions using a weight parameter and optimized the network simultaneously. Experimental results indicated that the proposed method enables joint analysis of acoustic events and scenes with reasonable performance compared with the conventional methods. Moreover, the proposed method improves the performance of acoustic event detection by 10.66 percentage points in terms of the F-score and by 0.081 in terms of ER, compared with a conventional CRNN-based method.

ACKNOWLEDGEMENT
This work was supported by JSPS KAKENHI Grant Number JP19K20304.