{'fraction_non_alphanumeric': 0.03313161875945537, 'fraction_numerical': 0.01758698940998487, 'mean_word_length': 4.614992567424082, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Panoramic Video Salient Object Detection with Ambisonic Audio Guidance

Video salient object detection (VSOD), as a fundamental computer vision problem, has been extensively discussed in the last decade. However, all existing works focus on addressing the VSOD problem in 2D scenarios. With the rapid development of VR devices, panoramic videos have been a promising alternative to 2D videos to provide immersive feelings of the real world. In this paper, we aim to tackle the video salient object detection problem for panoramic videos, with their corresponding ambisonic audios. A multimodal fusion module equipped with two pseudo-siamese audio-visual context fusion (ACF) blocks is proposed to effectively conduct audio-visual interaction. The ACF block equipped with spherical positional encoding enables the fusion in the 3D context to capture the spatial correspondence between pixels and sound sources from the equirectangular frames and ambisonic audios. Experimental results verify the effectiveness of our proposed components and demonstrate that our method achieves state-of-the-art performance on the ASOD60K dataset.


Introduction
Video salient object detection (VSOD) aims to find the most visually distinctive objects in a video. VSOD for 2D videos has been attracting considerable attention (Su et al. 2022;Liu et al. 2021;Ren et al. 2021b) due to its wide applications in real-world scenarios, such as video editing and video compression. While, for panoramic videos which have a very different format and viewing environment, how to effectively detect salient objects is still an open problem. Since human attention is usually influenced by acoustic signatures that are naturally synchronized with visual objects in audio-bearing video recordings, some VSOD methods for 2D videos (Tsiami, Koutras, and Maragos 2020;Cheng et al. 2021) introduce acoustic modality to facilitate the saliency discrimination. Different from mono or binaural audio used in 2D videos, ambisonic audio is utilized to create immersive feelings of the real world in panoramic videos. In this work, we focus on how to detect the salient objects in panoramic videos with their corresponding ambisonic audios. To the best of our knowledge, we are the first to tackle the VSOD problem for panoramic scenarios. Figure 1: We study the problem of how to utilize the ambisonic audio to facilitate the panoramic video salient object detection. Since ambisonic audio contains rich spatial information, we can directly localize the sound sources and then fuse them with visual cues. On the other hand, since the ER frame has distortions and cannot reflect the 3D position of pixels, projecting the pixels back to 3D space is essential for effective multimodal fusion.
The panoramic video contains omnidirectional contexts and represents each pixel on a 3D sphere. Unlike 2D videos that display with a stable viewpoint and fixed environmental audios, panoramic videos are typically supported by VR headsets which provide a head direction-adaptive field of view (FOV) and ambisonic audios. With multi-channel audio recordings, ambisonic audios encode the 3D location of the sound source and the VR headset can further adjust the original audio to provide a real sense of sound source to the user given the movement of the head. Previous works (Tsiami, Koutras, and Maragos 2020;Cheng et al. 2021) have demonstrated the audio shows non-trivial influence on human attention in 2D videos. For VSOD in panoramic videos, as shown in Figure 1, since the ambisonic audio can directly reflect the 3D sound source location, we consider it should perform a more important role compared to 2D scenarios.
On the other hand, previous panoramic video processing approaches cannot preserve the spatial relationship of panoramic videos. Due to the unique format of panoramic videos, it is difficult to store or transmit the raw video using current video coding techniques. Therefore, equirectangular (ER) projection is commonly leveraged to transform the panoramic video into a regular 2D format. However, the ER projection will involve not only a separation along the longitude of the sphere but also distortions in the polar area, which severely barrier the effective VSOD. Previous methods address the polar distortions by introducing a cube projection ) which projects the sphere to a cube and expands each face to ease the distortions. However, the padded cube map severely destroys the spatial relationship between each face which obstacles the global understanding of the frame.
In this paper, we purpose a framework for audio-visual salient object detection in panoramic scenarios. Considering the rich spatial and semantic information encoded in the ambisonic audios, we first use a pretrained acoustic encoder to extract location and semantic embeddings of 3D sound sources, and then introduce audio-visual context fusion (ACF) blocks to enhance the visual features by acoustic cues. Ideally, the 3D location of sound sources should be utilized as ground-truth to finetune the acoustic network while it is very difficult to obtain for common panoramic videos. To tackle this problem, inspired by label-guided distillation , the ground-truth label of salient object is employed in the multimodal fusion by enforcing consistency between teacher (equipped with label) and student branch. By learning better audio-visual correspondence, we can transfer the spatial information encoded in the visual objects to supervise acoustic modality. In particular, to reflect the true 3D location of objects and mitigate the influence of distortions in ER frames, we map the 2D coordinates of pixels back to the 3D sphere and encode them in the positional encoding during the fusion. In this way, the model can capture the true spatial position of each pixel. Our contributions are summarized as: • We propose an audio-visual video salient object detection framework for panoramic scenarios. To the best of our knowledge, we are the first to tackle this problem. • We introduce a label-guided audio-visual fusion module to effectively utilize the rich spatial and semantic information encoded in the ambisonic audio recordings synchronized with panoramic videos. • Our model achieves state-of-the-art results on the ASOD60K dataset. Extensive experiments are conducted to illustrate the effectiveness of our method.

Related Works
Video Salient Object Detection VSOD aims to find the most visually salient objects in the video. Conventional methods usually leverage color contrast (Achanta et al. 2009), motion prior (Zhang, Yu, and Crandall 2019), background prior (Yang et al. 2013) and center prior (Jiang and Davis 2013;Bertasius et al. 2017) to distinguish the salient regions. However, most of those methods are limited by the low representative ability of hand-crafted features. Recent methods leverage deep-learning approaches to tackle the VSOD problem. To utilize the temporal information, FCNS (Wang, Shen, and Shao 2017) first leverages FCN for static saliency prediction and then post-process them by another dynamic FCN. Similarly, DSRFCN3D (Le and Sugimoto 2017) introduces 3D convolution for temporal aggregation. Optical flow reflecting the motion also serves as a strong cue for VSOD task in (Li et al. 2018. Recently, more advanced structures are leveraged to better understand the spatio-temporal correspondence. For example, ConvL-STM (Li et al. 2018;Song et al. 2018;Fan et al. 2019) is adopted to construct long-term temporal relation. With the strong ability of transformer (Vaswani et al. 2017) to model the complex relationship, it has achieved promising result in VSOD (Liu et al. 2021;Ren et al. 2021b).

Panoramic Saliency Detection
Saliency detection aims to predict the region of human eye fixation in the video. For image-level saliency prediction Suzuki and Yamanaka 2018), ) propose a cube padding operation to project the panoramic frame to a cube with fewer distortions on each face. (Zhu, Zhai, and Min 2018) first map the ER frame to a sphere then predict the saliency from each viewport. For video-level saliency prediction Nguyen, Yan, and Nahrstedt 2018), Nguyen et al. (Nguyen, Yan, and Nahrstedt 2018;Zhang et al. 2018) proposes a method that leverages CNN and LSTM for saliency prediction. In (Zhang et al. 2018), spherical CNN is introduced to directly handle panoramic videos. In addition, audio is introduced in panoramic saliency prediction (Chao et al. 2020) given its strong ability to influence human attention.

Video Object Segmentation
Video object segmentation (VOS) can be categorized as unsupervised Ren et al. 2021a), semisupervised (Wang et al. 2021) and referring Li et al. 2022c) VOS. The most relevant type to this work is the unsupervised VOS (UVOS) which aims to segment primary object regions from the background in videos. Early methods tackle the UVOS problem by object proposal (Kim and Hwang 2002), temporal trajectory (Fragkiadaki, Zhang, and Shi 2012) and saliency prior . More recently, deep learning-based methods are proposed for modeling the spatio-temporal information. MATNet (Zhou et al. 2020) uses a motion-attentive transition to model motion cues and spatio-temporal representation. RTN (Ren et al. 2021a) leverages long-range intraframe contrast, temporal coherence, and motion-appearance similarity to enhance the appearance feature representation. In addition to VOS, video instance segmentation (Li et al. 2022a,b,d) is also relevant to this work. Recently, some works extend the video segmentation task to multimodal by considering audio (Zhou et al. 2022) or signals (Zhao et al. 2022;Huang et al. 2021b,a).

Method Overview
Given a video clip V = {I t } T t=1 of T frames and its corresponding multi-channel audio recordings A = {H i } 4 i=1 , we predict the salient object {M t } T t=1 effectively with our method. The method overview is illustrated in Figure 2. The pipeline can be boiled down into three parts: acoustic and visual encoders, a label-guided multimodal fusion module, and decoders. We first leverage a visual and an acoustic encoder to extract visual {f t } T t=1 and acoustic features g sem , g loc .

Ground Truth
Ground Truth ℒ !)*+& Figure 2: Pipeline Overview. We use separate encoders to extract multimodal features. For a video clip , a two-brunch acoustic encoder is employed to extract the semantic embedding g sem and location embedding g loc . After that, a label-guided multimodal fusion module is introduced to effectively fuse the multimodal features, which outputs a student feature {f stu t } T t=1 and a teacher feature {f tch t } T t=1 . Two decoders are leveraged to decode the final predictions {M stu t } T t=1 and {M tch t } T t=1 from compacted features {f stu t } T t=1 and {f tch t } T t=1 respectively. In particular, to enhance multimodal communication, a distillation loss L ditill is adopted between student feature {f stu t } T t=1 and (label-guided) teacher feature {f tch t } T t=1 . A structure loss L struc is utilized as the objective. The gray color indicates components that are only used during training.
The acoustic features contains a semantic embedding g sem and a location embedding g loc which encodes the category and 3D location of sound sources respectively. After that, a multimodal fusion module is utilized to enable audio-visual interaction. Specifically, the multimodal fusion module contains two pseudo-siamese blocks -a student block that fuses audio-visual information using multimodal attention and a teacher block that shares the same structure of the student block while taking an additional ground truth as input to guide the fusion. The output student features {f stu t } T t=1 and (label-guided) teacher features {f tch t } T t=1 are sent to visual decoders equipped with skip connections to generate the final salient object predictions {M stu t } T t=1 and {M tch t } T t=1 . A distillation loss between student feature {f stu t } T t=1 and (label-guided) teacher features {f tch t } T t=1 is adopted to help the multimodal interaction and a structure loss between pre-dictionM t and ground truth M t is used as the task objective for salient object detection.

Label-guided Multimodal Fusion
The label-guided multimodal fusion module contains two pseudo-siamese audio-visual context fusion (ACF) blocks equipped with spherical positional encoding to align the correspondence between 3D sound sources and pixels. The output of student and teacher block are student feature {f stu t } T t=1 and teacher feature {f tch t } T t=1 respectively. Spherical positional encoding. ER frame is a commonly used format to transmit and store panoramic videos (Cai et al. 2022). However, as shown in Figure 3, the ER frame suffers from severe distortions in the polar regions. To tackle this problem, we adopt the position-agnostic attention mechanism and propose a spherical positional encoding to compensate for the distortion in the ER frame. Dif- ferent from regular positional encoding (Vaswani et al. 2017), spherical positional encoding first re-projects the plane coordinates of each pixel back to the 3D sphere and generates positional encoding based on the 3D coordinates. In this way, each pixel can reflect its true 3D position thus avoiding the severe distortion in the polar region and inevitable separation along a longitude in the ER frame. In particular, the spherical positional encoding can be computed as PE(pos 3D , 2i) = sin(pos 3D /10000 2i/ d 3 ) and PE(pos 3D , 2i + 1) = cos(pos 3D /10000 2i/ d 3 ) where pos 3D can be x, y, z coordinates and i is the dimension. The 2D-to-3D transformation can be computed as where (u, v) and (x, y, z) are the 2D and 3D coordinate of each pixel respectively. R = W 2π where W is the width of the frame. Spherical positional encoding (SPE) is employed to encode spatial information for visual representation during cross-modal attention.
Student block. As shown in Figure 4 (without the gray parts), to fuse the rich information encoded in visual and acoustic features, we utilize multimodal attention to enable audio-visual context interaction. We first concatenate  Figure 4: Illustration of ACF block used in the (label-guided) multimodal fusion process. Student block: The visual feature {f t } T t=1 is first flattened and added with positional encoding then conducts multimodal attention with acoustic semantic embedding g sem and location embedding g loc respectively. After that, we form a pixel-wise weighting from fused feature F loc to modulate F sem . The output of student block is denoted as {f stu t } T t=1 . Teacher block: Different from student block, we concatenate ground-truth mask with the visual feature {f t } T t=1 to help the network learn better representation in the multimodal fusion. Since teacher block is only employed during training, we truncate the gradients before the multimodal fusion in the teacher block. The output of teacher block is denoted as {f tch t } T t=1 .
and then flatten the visual feature {f t } T t=1 to form F = f latten(f 1 ⊕ · · · ⊕ f T ) ∈ R C×T HW . After that, spherical and regular positional encodings (Vaswani et al. 2017) are added to visual feature F , and acoustic feature g sem and g loc , respectively, to help the network capture spatial information. The multimodal attention can be computed by where MCA, FFN and LN are multi-head cross-attention (Ye et al. 2019), feed-forward network and layernormalization respectively. In particular, we generate the query Q from F and key K, value V from g aud by linear projections. The MCA(F, g aud ) can be computed by where d is the dimension of query Q. The student feature F stu can be computed by where ϕ C →1 denotes a convolution to reduce channel from C to 1 and denotes element-wise multiplication. The final output is {f stu t } T t=1 = Reshape(F stu ). Teacher block. The audio encoder is pretrained on a 3D sound source localization dataset (Guizzo et al. 2021) while it is difficult to obtain a 3D location of vocal objects in panoramic videos during main training. Inspired by previous work ), we build a pseudo-siamese teacher block to help the network capture accurate spatial information by introducing ground-truth annotation to the multimodal fusion. As shown in Figure 4 (with the gray parts), the additional ground truth is downsampled and concatenated with visual feature {f t } T t=1 in the teacher block before conducting the multimodal interaction. Similar to student block, the final output of teacher block is denoted as {f tch t } T t=1 . Note that the teacher block does not share weights with the student block and the gradient of teacher block is truncated to avoid influencing the encoder.

Encoder
We use separate encoders to extract visual and acoustic features.
Visual Feature ! " Frame ! Transformer Backbone F P Addition P Positional Encoding F Flatten Figure 5: Single-frame visual feature extraction. The ER frame I t is first processed by a backbone and then the extracted features are fed to a transformer to capture spatial information. The single-frame feature is denoted as f t .
Visual encoder. We first project the panoramic video to 2D frames {I 1 · · · I T } using ER projection and then feed them to the backbone. As shown in Figure 5, a transformer encoder on top of the ResNet-50 (He et al. 2016) is adopted to mitigate the severe distortion in the ER frame. In addition, a temporal non-local block as (Yan et al. 2019) is also leveraged to enable the temporal interaction on the extracted features {f t } T t=1 from the backbone. We denote the features after temporal aggregation as {f t } T t=1 where f t ∈ R C×H×W . Acoustic encoder. Multi-channel audio contains the 3D location and category information of the sound source which have a great impact on the choosing of the salient objects. To extract the acoustic features, we leverage three CNN layers followed by two bi-directional GRU layers as our acoustic encoder and three linear layers for both semantic and location head (detailed structure available in supplementary) (Adavanne et al. 2018). Since it is difficult to obtain the realworld sound source location in panoramic videos, we first pretrain the acoustic encoder on a 3D sound source localization and sound event classification dataset, L3DAS (Guizzo et al. 2021). We remove final linear layer in each head to form the semantic embedding g sem ∈ R C×L and location embedding g loc ∈ R C×L .

Decoder
We adopt the same structure for decoding {f stu t } T t=1 and {f tch t } T t=1 . For decoding the salient object prediction, we follow the FPN structure (Lin et al. 2017) to fuse the lowlevel features. Let the output salient object prediction be {M stu t } T t=1 ∈ R Ho×Wo and {M tch t } T t=1 ∈ R Ho×Wo for student and teacher branch respectively, where H o and W o are the height and width of the output.

Loss Function
The overall objective of our proposed method is composed of structure losses for student and teacher branch L stu struc , L tch struc and a distillation loss L distill L = L stu struc + L tch struc + λ distill L distill (5) where λ distill is a scalar to balance the losses.
Structure loss. Following previous method (Chen et al. 2022), we leverage a combination of binary cross entropy loss and Dice loss (Milletari, Navab, and Ahmadi 2016) as the objective for salient object detection.
where M t andM t are predicted and ground-truth salient maps respectively. λ dice is a scalar. M t can be M stu t and M tch t for computing L stu struc and L tch struc . Distillation loss. To help the student block learn the audiovisual correspondence, we enforce a consistency between f tch t and f stu t . A MSE loss is utilized as the constraint

Inference
Since the purpose of introducing teacher block is to facilitate network learn accurate audio-visual correspondence during training, we disable the teacher block and only keep the student block during inference.

Experiment Dataset and Metrics
Dataset. We conduct experiments on the ASOD60K dataset (Zhang, Chao, and Zhang 2021) which is an audio-induced salient object detection benchmark for panoramic videos. There are 62,455 frames with 10,465 instance-level ground truths in the dataset. In particular, each video corresponds to a 4-channel ambisonic audio recording. The ground-truth salient objects are determined by the eye fixation of 40 participants who viewed the video with HTC Vive HMD headset. The test set of ASOD60K contains three subsets split by sound event classes -miscellanea, music, and speaking.
Metrics. To evaluate the performance of video salient object detection, we employ the adaptive F-Measure

Implementation Details
Following the benchmark setting (Zhang, Chao, and Zhang 2021), our model is first pre-trained on DUST dataset ) and then finetuned on ASOD60K (Zhang, Chao, and Zhang 2021). The model is trained for 20 epochs with a learning rate of 1e-4. We adopt a batchsize of 2 and an AdamW (Loshchilov and Hutter 2017) optimizer with weight decay 0. All images are cropped to have the longest side of 832 pixels and the shortest side of 416 pixels during training and evaluation. The window size is set to 3. The λ distill is set to 5.0 and λ dice is set to 1 if no specification. We leverage a 3-layer transformer encoder (Dosovitskiy et al. 2020) on top of the ResNet-50 (He et al. 2016) to extract visual features. We leverage an augmented SELDNet (Adavanne et al. 2018) as our acoustic encoder. Our method is implemented with PyTorch.

Main Results
In this section, we compare our method with previous stateof-the-art methods, including CPD-R (Wu, Su, and Huang 2019a), MINet , SCRN (Wu, Su, and Huang Quantitative results. We compare our method with stateof-the-art methods on the ASOD60K dataset in  Qualitative results. We present our qualitative result in Figure 6 and compare it against previous 2D methods on ASOD60K dataset. The result indicates that previous methods fail to detect the correct salient objects. In contrast, our method shows great accuracy and robustness even in very challenging scenarios, e.g., with severe distortions in the polar area as shown in the first line in Figure 6. This implies that our network equipped with ACF block and SPE generates more accurate results than simply adopting previous 2D methods on the panoramic scenario. We visualize the audioguided location heatmap ϕ C →1 (f loc t ) in Figure 7. We notice that the audio-guided location heatmap ϕ C →1 (f loc t ) reflects the correct location of sound sources thus helping the final salient object detection.

Ablation Experiments
We conduct extensive ablation studies on the ASOD60K dataset to verify the effectiveness of different components.      Multimodal fusion method. To investigate the effectiveness of our proposed ACF block, we conduct experiments with different multimodal fusion schemes. As shown in Table 2, we compare the ACF block equipped with multimodal attention with two baseline settings. 'None' and 'ACF (Concat)' means no multimodal fusion and simply concatenating visual and acoustic features in the ACF block respectively. In general, the model without multimodal fusion leads to an inferior performance which indicates the acoustic modality is essential to salient object detection. We notice that ACF block equipped with multimodal attention outperforms the ACF block with simply multimodal feature concatenation. Additional spherical positional encoding (SPE) brings another 0.07 F β , 0.08 S α and 0.1 E φ gain compared to multimodal attention.
Visual feature extraction. We conduct experiments to ablate the influence of different visual feature extraction methods. We first build a baseline model that leverages ResNet-50 (He et al. 2016) backbone to extract visual features which leads to 0.396 F β , 0.660 S α , 0.714 E φ and 0.37 M. By employing a transformer encoder on top of the backbone, we observe non-trivial gains on all metrics. We consider the improvement comes from the strong global understanding capability of transformer. By replacing the standard positional encoding in the transformer with our spherical positional encoder, the E φ metric improves 0.1 while F β and S α slightly drop. We consider the performance drop mainly because the pretraining is conducted on the 2D dataset.
Distillation loss weight. To investigate the influence of distillation loss weight, we conduct experiments by ablating different λ distill . As shown in Table 4, we notice that a weight of 5 leads to the best result of 0.404 F β , 0.678 S α , 0.732 E φ and 0.26 M. The λ distill = 0 means that no teacher branch is adopted which leads to the worst result.
3D sound source localization. To demonstrate the effectiveness of employing 3D sound source localization in VSOD, we conduct an experiment to disable the sound source localization branch in our method. The result in Table 5 indicates that 3D sound source localization from ambisonic audio can help salient object detection in panoramic scenarios.
Window size. Since temporal information is essential for the video salient object detection, we conduct experiments on different input window sizes as shown in Table 6. We notice that the window size of 3 achieves the best performance in terms of F β , E φ and M and the window size of 4 achieves the best result in terms of S α .
Sound type. We conduct experiments to show the benefit of utilizing ambisonic audios. Table 7 shows that mono audio only has a trivial improvement compared to the baseline setting (no multimodal fusion). We consider this is because mono audio cannot explicitly model spatial information of the sound source.

Conclusion
In this paper, we propose a framework for audio-visual video salient object detection in panoramic scenarios. In particular, we propose an audio-visual context fusion block to enhance visual features by ambisonic audios. To better utilize the spatial information encoded in the audios, a label-guided distillation scheme is introduced to help the multimodal interaction. In addition, due to the severe distortions in the ER frame, we leverage position-agnostic attention mechanism equipped with spherical positional encoding to map each pixel back to 3D space thus capturing the true spatial location of each pixel. Notably, our method achieves the best result on the ASOD60K benchmark. Moreover, extensive study shows that ambisonic audio can help the salient object detection in panoramic videos.