{'fraction_non_alphanumeric': 0.038401528419041556, 'fraction_numerical': 0.01614392612641299, 'mean_word_length': 4.71122022185852, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Visually Guided Sound Source Separation using Cascaded Opponent Filter Network

The objective of this paper is to recover the original component signals from a mixture audio with the aid of visual cues of the sound sources. Such task is usually referred as visually guided sound source separation. The proposed Cascaded Opponent Filter (COF) framework consists of multiple stages, which recursively refine the sound separation based on appearance and motion information. A key element is a novel opponent filter module that identifies and relocates residual components between sound sources. Finally, we propose a Sound Source Location Masking (SSLM) technique, which, together with COF, produces a pixel level mask of the source location. The entire system is trained end-to-end using a large set of unlabelled videos. We compare COF with recent baselines and obtain state-of-the-art performance in three challenging datasets (MUSIC, A-MUSIC, and A-NATURAL). The implementation and pre-trained models will be made publicly available.


Introduction
Sound source separation [17,36,8,42] is a classical audio processing problem, where the objective is to recover original component signals from a given mixture audio. Well known example of such task is the cocktail party problem, where multiple people are talking simultaneously (e.g. at a cocktail party) and the observer is attempting to follow one of the discussions. The general form of the problem is challenging and highly underdetermined. Fortunately, one is often able to leverage additional constraints from external cues, such as vision. For instance, the cocktail party problem turns more tractable by observing the lip movements of people [12]. Similar visual cues have also been applied in other sound separation tasks [13,31,49,48,45,14,15]. This type of problem setup is ofter referred as visually guided sound separation (see e.g. Fig. 1).
Besides separating the component signals from the mixture, one is often interested in identifying the source locations. Such task would be intractable from a single audio channel, but could be approached using e.g. microphone arrays [34]. Alternatively, the sound source location can be determined from the visual data [40,22]. Such task requires learning the correlations between audio and the appearance or motion of the source (e.g. instrument). However, it would be costly to obtain labelled examples for learning the necessary representations. Fortunately, plausible audio mixtures can be created by artificially summing . The proposed COF approach results in better separation performance over the baseline methods SoP [49], SoM [48], and MP-Net [45].
up audio signals. Furthermore, a large number of videos with naturally aligned audio is available at the Internet. These facts have inspired several works to present self-supervised representation learning approaches for multiple tasks [4,29], including visually guided source separation [13,31,49,48,45,14] and sound source localisation [2,3,40,22]. This paper proposes a new approach for visually guided sound source separation and localisation. Our system (Fig. 2) consists of cascaded stages of Opponent Filter (OF) modules (Fig. 5a), which use visual features of a sound source to look for incorrectly assigned sound components from other sources and reassigns them (by removing them from the current source and adding to the first source). We show that the OF module can greatly improve the source separation performance compared to the recent single stage systems [49,48] and recursive approach [45]. Moreover, since motion is strongly correlated to sound formation [48], we build our system on both appearance and motion representations. To this end, we examine multiple options based on RGB frames, optical flows, dynamic images [5], and their combinations. Finally, we introduce a Sound Source Location Masking (SSLM) network that, in conjunction with COF, is able to pin point pixel level segmentation of the sound source location. Qualitative results indicate sharper and more accurate results compared to the baselines [49,48,45]. The entire system is trained using a self-supervised setup with large set of unlabelled videos.

Related Work
Cross-modal Learning from Audio and Vision Aytar et al. [4] presented a method for learning joint audio-visual embeddings by minimizing the KLdivergence of their representations. Owens et al.
[32] obtained supervision sound for visual representation learning. Arandjelovic et al. [2,3] associated the learnt audio and visual embeddings by asking whether they originate from the same video. Nagrani et al.
[29] learned to identify face and voice correspondences. More recent works, include transferring mono-to binaural audio using visual features [14], audio-video deep clustering [1], talking face generation [50], audio-driven 3D facial animation prediction [9], and speech embedding disentanglements [30]. Unlike these works, (visually guided) sound source separation aims at splitting the input audio into original components signals.
Video Sequence Representations Most early works in video representations were largely based on direct extensions of the image based models [27, 43,25]. More recently, these have been replaced by deep learning based alternatives operating on stack of consecutive video frames. These works can be roughly divided into following categories: 1) 3D CNN applied on spatio-temporal video volume [41]; 2) two-stream CNNs [37,6,47] applied on video frames and separately computed optical flow frames; 3) LSTM [11], Graph CNN [44] and attention clusters [28] based techniques; and 4) 2D CNN with the concept of dynamic image [5]. Since most of these methods are proposed for action recognition problem, it is unclear which representation would be best suited for self-supervised sound source separation. Therefore, this paper evaluates multiple options and discusses their pros and cons.
(Visually Guided) Sound Source Separation The sound source separation task is extensively studied in the audio processing community. Early works were mainly based on probabilistic models [17,36,8,42], while recent methods utilise deep learning architectures [38,7,20,18]. Despite of the substantial improvements, the pure audio based source separation remains a challenging task. At the same time, visually guided sound source separation has gained increasing attention. Ephrat et al. [12] extracted face embeddings to facilitate speech separation. Similarly, Gao et al. [15,13] utilised object detection and category information to guide source separation. While impressive, these methods rely on the external knowledge of the video content (e.g. speaking faces or object types).
The works by Zhao et al. [49,48] and Xu et al. [45] are most related to ours. In [49] the input spectrogram is split into components using U-Net [35] architecture and the separated outputs are constructed as a linear combinations of these. The mixing coefficients are estimated by applying Dilated ResNet to the keyframes representing the sources. The subsequent work [48] introduced motion features and improvements to the output spectrogram prediction. Both of these methods operate in a single stage manner directly predicting the final output. Alternatively, Xu et al. [45] proposed to separate sounds by recursively removing large energy components from the sound mixture. Differently, our work explores multiple approaches to utilize the appearance and motion information based on RGB frames, optical flows, and dynamic images to refine the sound source separation in multi-stages. Our proposed Opponent Filter uses visual features of a sound source to look for incorrectly assigned sound components from opponent sources, resulting in accurate sound separation.  [45] visualise the sound sources by calculating the sound volume at each spatial location. In contrast to these methods, which either produce coarse sound location or rely on the external knowledge, we propose a self-supervised SSLM network to localise sound sources on a pixel level.

Methods
This section describes the proposed visually guided sound source separation method. We start with a short overview and then continue to detailed descriptions of each component.

Overview
The inputs to our system consist of a mixture audio (e.g. band playing) and a set of videos, each representing one component of the mixture (e.g. person playing a guitar). The objective of the system is to recover the component signals corresponding to each video sequence. Fig. 2 illustrates an overview of the approach. Note that the audio signals are represented as spectrograms, which are obtained from the audio stream using Short-term Fourier transform (STFT).
The proposed system consists of multiple cascaded stages. The first stage contains three components: 1) a sound network that splits the input spectrogram into a set of feature maps; 2) a vision network that converts the input video sequences into compact representations; and 3) a sound separator that produces spectrograms of the component audios (one per video) based on the outputs of the sound and vision networks. The second stage contains similar sound and vision networks as the first one (internal details may differ). However, instead of the sound separator, the second stage contains a special opponent filter (OF) module, which enhances the separation result by transferring sound components between the sources. The output of the filter is passed to the next stage or used as the final output. The following stages are identical to the second one and, for this reason, we refer our method as cascaded opponent filter (COF) network. The final component audios are produced by applying the inverse STFT to the component spectrograms.
In addition, we propose a new Sound Source Location Masking (SSLM) network (not shown in Fig. 2) that indicates the pixels with highest impact on the sound source separation (i.e. source location). The entire network is trained in end-to-end fashion using artificially generated examples. That is, we take two or more videos and create an artificial mixture by summing the corresponding audio tracks. The created mixture and video frames are provided to the system, which then has to reproduce the original component audios. In the following sections, we will present each component with more details and provide the learning objective used in the training phase.

Vision Network
The vision network aims at converting the input video sequence (or keyframe) into a compact representation that contains the necessary information of the sound source. Sometimes already a pure appearance of the source (e.g. instrument type) might be sufficient, but, in most cases, the motions are vital cues to facilitate the source separation (e.g. movements of the players hand, mouth motion, etc.). To this end, we study several visual representation options described in the following. In all cases, we assume that the input video sequence is of size 3 × 16H × 16W and has T frames. The detailed network architectures are provided in the supplementary material.
The first option, referred as C2D-RGB, is a pure appearance based representation. This is obtained by applying a dilated ResNet18 [19] to a single keyframe extracted from the sequence. More specifically, given an input RGB image of size 3 × 16H × 16W, the C2D-RGB produces a representation of size K × H × W. Dynamic image [5] is a compact representation, which summarises the appearance and motion of the entire video sequence into a single RGB image by rank pooling to the original pixel data. In the second option, referred as C2D-DYN, we first convert the input video into a dynamic image (size 3 × 16H × 16W) and then apply a dilated ResNet18 [19] to produce a representation of size K × H × W. Fig. 3c illustrates C2D-DYN option.
The third option, referred as C3D-RGB, applies 3D CNN to extract the appearance and motion information from the sequence simultaneously. C3D-RGB uses 3D version of ResNet18 and produces a representation of size T × K × H × W. The optical flow [37,39,24] explicitly describes the motion between the video frames. In the fourth option, referred as C3D-FLO, we first estimate the optical flow between the consecutive video frames using LiteFlowNet [24], and then apply 3D ResNet18 to the obtained flow sequence. C3D-FLO produces a representation of size T × K × H × W.
In addition, following the recent work [6] in action recognition, we propose a set of two stream options by combining pairs of C2D-RGB, C3D-RGB, and C3D-FLO representations using Mutual Attention (MA) module. The module is depicted in Fig. 3d and it enhances the sound source relevant motions by multiplying the C3D features with a spatial attention map. The appearance-weighted features are added back to the original C3D features in order to keep C3D features as the principle cue in case the C2D-RGB fails to localize the sound source. We obtain C3D feature attention by adding a sigmoid function on top of the final enhanced C3D features. The multiplication between the C3D feature attention and the time-inflated appearance features are added back to the C2D-RGB appearance features. Within this process, for the predicted regions of interest from C2D-RGB, the appearance that has no motions will be eliminated. Finally, we receive the mutual attentive features of dimension T × K × H × W from the two-stream structures, which are referred to as MA(C2D-RGB, C3D-RGB) and MA(C2D-RGB, C3D-FLO). Fig. 3a and 3b illustrate these options. We omit the model of two 3D streams MA(C3D-RGB, C3D-FLO) due to large size of the resulting model.

Sound Network
The sound network splits the input audio spectrogram into a set of feature maps. The network is implemented using U-Net [35] architecture and it converts the input spectrogram of size HS × WS into an output of size HS × WS × K. Note that the number of created feature maps K is equal to the visual feature dimension K in the previous section. At the first stage, the input to the sound network is the original mixture spectrogram X mix , while in later stages, the sound network operates on the current estimates of the component spectrograms. This allows stages to focus on different details of the spectrogram. In the following, we will denote the kth feature map, produced by the sound network for an input spectrogram X, as S(X) k .

Sound Separator
The sound separator combines the visual representations with the sound network output and produces an estimate of the component signals. We introduce two different approaches for this part and refer them as Global Sound Separator (GSS) and Temporal Sound Separator (TSS). The GSS option is used for all experiments, except for Section 4.5.
Global Sound Separator First, we apply global max pooling operation over the spatial dimensions (H × W) of the visual representation. For 3D CNN based options, we further apply max pooling layer along the temporal dimension T . As a result, we obtain a feature vector z with K elements. We combine z with sound network output using a linear combination as follows where α k and β are learnable weight parameters, z k is the kth element of z, and S(X) k is the kth sound network feature map for a spectrogram X.  of size T × K. Furthermore, we split the sound network output in T slices along the temporal (WS) dimension. This operation results in a sound feature map of size T × K × HS × WS T . Finally, we combine these two representation as where z k,i denotes the kth element in the ith temporal feature vector, S(X) k,i is the ith temporal slice of the kth feature channel of the sound network output, α k,i and β i are parameters. The process is illustrated in Fig. 4b. We perform evaluations on TSS in Section 4.5.

Component Spectrograms
The produced results are further processed into three outputs as (similarly for TSS) where σ denotes the sigmoid function, th represents the thresholding operation with value 0.5, and ⊗ is the element-wise product. In other words, we first map g into a binary maskb, and then produce the estimate of the output component spectrogram as an element-wise multiplication between the mask and the original mixture spectrogram X mix . g andŶ are provided for the upcoming stage as inputs (or used as the final output). We will denote the outputs corresponding to nth video at stage j as [g n ] j , [b n ] j , and [Ŷ n ] j .

Opponent Filter Module
We depict the idea of Opponent Filter (OF) module using an example case of two videos in Fig. 5a. OF uses visual features of a sound source to look for incorrectly assigned sound components from other opponent sources and reassign them by adding to the first source and removing from corresponding opponent sources. More specifically, where GSS is as defined in Eq. (1), z n is the visual representation of nth video, m denotes the mth audio component, and ⊕ and denote the element-wise sum and subtraction, respectively. GSS(z n , [Ŷ m ] j−1 ) denotes the sound components selected by the visual cues of nth video from mth opponent sound. It will be added to the nth output of previous stage [g n ] j−1 , and be subtracted from the opponent predictions from previous stage [g m ] j−1 . We apply the OF module to the cascaded network since stage 2.

Learning Objective
The model parameters are optimised with respect to the binary cross entropy (BCE) loss that is evaluated between the predicted and ground truth masks over all stages. More specifically, where r j is a weight parameter, [b] j is the predicted mask, b gt is the ground truth mask, and J is the total number of stages.

Sound Source Location Masking Network
The objective of the Sound Source Location Masking (SSLM) network is to identify a minimum set of input pixels, for which the COF network would produce almost identical output as for the entire image. In practice, we follow the ideas presented in [23], and build an auxiliary network to estimate a sound source location mask that is applied to the input RGB frames. We illustrate the overall structure of the SSLM in Fig. 5b. The SSLM network is implemented using a dilated residual network (DRN) [46] pre-trained on ImageNet [10], with three up-projection blocks [26] followed by a  Table contains three where l 1 denotes the L 1 norm, [b SSLM ] j is the output sound separation mask obtained using only selected pixels, [b] j is the output separation mask for the original image, r j and λ are weight parameters. SSLM(I) is the output sound source location mask, and q is its total number of pixels.

Experiments
We evaluate the proposed approach using Multimodel Sources of Instrument Combinations (MUSIC) [49] dataset, and two sub-sets of AudioSet [16]: A-MUSIC and A-NATURAL. The proposed model is trained using artificial examples, generated by adding audio signals from two of more training videos. We refer to our method as COF(v 1 , v 2 , . . .), where v j is substituted by the visual representation used in stage j. For example, COF(C2D-RGB,C3D-RGB) refers to a configuration where the first and second stages apply C2D-RGB and C3D-RGB, respectively. The performance of the final sound source separation is measured in terms of standard metrics: Signal to Distortion Ratio (SDR), Signal to Interference Ratio (SIR), and Signal to Artifact Ratio (SAR). We explain the datasets and implementation details in the supplementary materials. In the following sections, we evaluate different parts of the proposed system and provide a comparison with the state-of-the-art methods.

Opponent Filter
In this section, we assess the performance of the opponent filter module. For simplicity, we perform these experiments using only the appearance based features C2D-RGB. The baseline is provided by the basic single stage version COF(C2D-RGB), which does not contain OF module. The results are provided in Table 1, which indicates that the two stage version with the OF module clearly outperforms the baseline. In addition, we evaluate the impact of the "addition" and "subtraction" branches in the OF module. To this end, we implement two versions COF addition and COF subtraction , which include only the "addition" and "subtraction" operation in the OF, respectively. The corresponding results in Table 1 indicate that both versions obtain similar performance which is between the baseline and the full model. We conclude that both operations are essential part of the OF module and contribute equally to the sound separation result.

Visual Representations
We firstly separate sounds by implementing a single stage network with GSS, associating with appearance and motion cues that are extracted from the discussed two-stream C3D-RGB, two-stream C3D-FLO, and C2D-DYN. We denote the two-stream C3D-RGB and C3D-FLO as MA-RGB and MA-FLO in Table 2. As is shown in the block 1 of Table 2, the results with appearance and motion cues clearly surpass the network with only appearance cues from C2D-RGB, which proposes that the motion representation is important for the sound separation quality. Block 2 of Table 2 shows the performance of how the visual information separates sounds in a two-stage manner. Explicitly, we replace the vision network at each stage in Fig. 2 with MA-RGB, MA-FLO, and C2D-DYN. Table 2 reports that the three two-stage networks obtain similar performance and outperform their single-stage counterparts from block 1 with a large margin. Table 3. The performance of sound separation of our 2-stage and 3-stage COF models in comparison to three recent baselines SoP [49], SoM [48], and MP-Net [45], on MUSIC, A-MUSIC, and A-NATURAL datasets. The ". . . " represents C2D-DYN Finally, we evaluate an option where the first stage utilises only appearance based option and the second stage applies motion cues. In practice, we combine C2D-RGB with C3D-RGB, C3D-FLO, or C2D-DYN. The results in Table 2, indicate that this combination obtains similar or even better performance than the options where motion information was provided for both stages. We conclude that the appearance information is enough to facilitate coarse separation at first stage. The motion information is only needed at the later stages to provide higher separation quality. It is worth noting that the COF(C2D-RGB, C2D-DYN) combination has less parameters and equal performance compared to the 3D CNN alternative. Therefore, we chose this model for the later experiments with larger datasets.

Visualizing Sound Source Locations
We compare the sound source localizing capability of our best two-stage model COF(C2D-RGB, C2D-DYN) with state-of-the-art methods in Fig. 6. Columns (b)-(e) display the sound energy distributions of spatial location in heatmaps on input frame during inference. COF produces precise associations between visual representation and separated sounds, though columns (e) is just the visualization from the first stage of COF. As we know, the spatial features from ConvNet with stride=16 usually have small resolution (14 × 14 pixels in this work). Thus, the final visualized location is generally coarse after up-sampling the heatmap to the resolution of the input image. Differently, our proposed SSLM learns to predict a pixel-level sound source location mask, as shown in column (f), which precisely localizes sound sources and preserves high quality of sound separation. Further examples are provided in the supplementary material.

Comparison to State-of-the-Art
Here we compare two versions of the proposed model with three recent baseline methods SoP [49], SoM [48], and MP-Net [45] for visually guided sound source separation. The corresponding results for MUSIC, A-MUSIC, and A-NATURAL datasets are provided in Table 3 and Fig. 7. The quantitative results indicate  [49], SoM [48], and MP-Net [45].
that our model outperforms the baselines with a large margin across all three datasets.
To verify the scalability of our method on the sound mixture of more sounds, we train and evaluate our two-stage model COF(C2D-RGB, C2D-DYN) on the mixture of 3 videos on MUSIC dataset. It achieves the performance of SDR: 3.33, SIR: 10.32, and SAR: 6.70 compared to the SDR: 1.30, SIR: 8.66, and SAR: 5.73 of MP-Net. To study the generalization of our proposed COF network in the aspect of data variety and stage extension, we train the method on the three datasets with 2 and 3 stages. We observe from Table 3 that when increasing the number of stages, the performance generally improves until reaching the plateau.

Motion Representation Evaluations
The final experiment highlights the importance of the temporal variance provided by TSS (see Sec. 3), we train, evaluate, and test the network COF(MA-RGB, MA-RGB) on Aug(0), Aug(1) and Aug(2) versions of the MUSIC dataset. Aug(0) corresponds to the original MUSIC dataset, from which sound mixture is formed by videos of different instrumental categories. Aug(1) randomly selects videos from same categories, and in Aug(2), except constructing sound mixture of using videos of same categories, we randomly "mute" 1 second of each video by replicating its first frame to all the rest frames within this second and setting the 1 -second audio to zeros. The results in Table 4 indicate that GSS performs better than TSS on the Aug(0). However, both models fail to separate sounds on the hardest case of Aug(1). The coinstantaneous motions from two videos of the same instrument are not sufficient to separate sound successfully. We hypothesize that if motion is the only variable at some point, the COF model with TSS is capable of assigning the sound components from mixture of same category to the video that has motions, and further learns to distinguish the motion differences of videos. We implement the hypothesis on Aug(2) in Table 4. The model with TSS surpasses the counterpart models of GSS and SoM.

Conclusions
We proposed an innovative framework of visually guided Cascaded Opponent Filter (COF) network for sound source separation. In contrast to recent methods, COF recursively refines the sound separation by utilizing both the appearance and motion information in multi-stages. For this purpose, we evaluated several representations based on RGB frames, optical flows, and dynamic image. The Opponent Filter (OF) module boosts the sound separation quality by identifying and relocating residual components between sound sources. Besides, we introduced a Sound Source Location Making (SSLM) network, together with COF, to precisely localize sound sources. These novel contributions delivered significantly improved sounds separation and accurate sound source localisation. We have performed extensive evaluations on our proposed methods and obtained state-ofthe-art performance on challenging datasets. In this section, we present sound source separation performance for different instrument mixtures using MUSIC dataset. Fig. 8 illustrates the results in terms of SDR in a matrix form. The diagonals represent the results of separating instruments of same categories (e.g. two guitars), and the off-diagonals are combinations from different categories (e.g. guitar and violin). The higher value of SDR represents the better performance of sound source separation, e.g. acoustic guitar and flute (2 and 6), flute and tuba (6 and 9), tuba and xylophone (9 and 11). The SDR values on the diagonals clearly indicate that separating sounds between two instruments from the same category is the hardest case. In particular, separating the mixture of two flutes is challenging. One reason might be the small amount of motion related to playing flute. The SDR values on the diagonals clearly indicate the hardest case of separating sounds between two instruments from the same category, e.g. flute (6 and 6).

A.2 Sound Source Localization Examples
We visualize more examples of localized sounding sources by our proposed Sound Source Location Masking (SSLM) network in comparison with baseline methods of SoP [49], SoM [48], and MP-Net [45] on MUSIC, A-MUSIC and A-NATURAL datasets in Fig. 9, Fig. 10, and Fig. 11 respectively.

A.3 Datasets
We evaluate the proposed approaches using Multimodal Sources of Instrument Combinations (MUSIC) [49] dataset, and two sub-sets of AudioSet [16]: A-MUSIC and A-NATURAL. A-MUSIC and A-NATURAL AudioSet consists of an expanding ontology of 632 audio event classes and is a collection of over 2 million 10-second sound clips drawn from YouTube videos. Many of the AudioSet videos have limited quality and sometimes the visual content might be uncorrelated to the audio track. A-MUSIC dataset is a trimmed musical instrument dataset from AudioSet. It has around 25k videos spanning 10 instrumental categories: accordion, bagpipe, cello, flute, piano, pizzicato, saxophone, trumpet, ukulele, and zither. A-NATURAL dataset is a trimmed natural sound dataset from AudioSet. It contains around 10k videos which cover 10 categories of natural sounds, namely baby crying, chainsaw, dog, drum, firework, helicopter, printer, rail, snoring, and water. We split both the A-MUSIC and A-NATURAL dataset samples to 80%, 10%, and 10% as train, validation and test set.

A.4 Implementation Details
Network Architecture Details We extract video frames at 8fps and adopt frame augmentation by random scaling, random horizontal flipping, and random cropping (224 × 224) during training for all datasets. We apply a dilated 2D ResNet18 [19] with dilation=2 to obtain representations of C2D-RGB and C2D-DYN. For a single input RGB image or dynamic image of size 3 × 16H × 16W, we truncate the ResNet18 after stride=16 and achieve the visual feature of size K × H × W by performing a 3 × 3 convolution with output channels of K =16 on the top. The C3D models utilize 3D version of ResNet18 on T =48 frames. With   the stride=16 on spatial dimension and stride=8 on the temporal dimension, we yield the C3D-RGB and C3D-FLO representations of size T × K × H × W, where T =6 and H = W = 14. In the Mutual Attention (MA) module, we obtain the spatial attention map by projecting the appearance features from C2D-RGB to a single-channel feature map with a 1 × 1 convolution and a sigmoid operation. We sub-sample each audio signals at 11kHz and randomly crop an audio clip of 6 seconds for training. A Time-Frequency (T-F) spectrogram of size 512 × 256 is obtained by applying STFT, with a Hanning window size of 1022 and a hop length of 256, to the input sound clip. We further re-sample this spectrogram to a T-F representation of size 256 × 256 on a log-frequency scale and feed it to the Sound Network S. We adopt the U-Net [35] with 7 layers of 2D CNN and output channels of K=16 as the architecture of Sound Network. To obtain the final separated audio signals, the inverse STFT is applied to the component spectrograms.
Optimization Our implementation is built on Pytorch. The network is trained with a batch size of 10 for 4,000 iterations. We use stochastic gradient descent (SGD) with momentum 0.9 and weight decay 1e-4 to train our Cascaded Opponent Filter (COF) network and Adam optimizer to train the Sound Source Location Masking (SSLM) network. The vision networks of COF and the SSLM, pretrained on ImageNet [10], use a learning rate of 1e-4, while the rest of modules which are trained from scratch use a learning rate of 1e-3. We decrease the learning rate from its initial value by a factor of 10 every 1,600 iterations.