{'fraction_non_alphanumeric': 0.03714188100932207, 'fraction_numerical': 0.015349699838937967, 'mean_word_length': 4.79782116581777, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Attention Bottlenecks for Multimodal Fusion

Humans perceive the world by concurrently processing and fusing high-dimensional inputs from multiple modalities such as vision and audio. Machine perception models, in stark contrast, are typically modality-specific and optimised for unimodal benchmarks, and hence late-stage fusion of final representations or predictions from each modality (`late-fusion') is still a dominant paradigm for multimodal video classification. Instead, we introduce a novel transformer based architecture that uses `fusion bottlenecks' for modality fusion at multiple layers. Compared to traditional pairwise self-attention, our model forces information between different modalities to pass through a small number of bottleneck latents, requiring the model to collate and condense the most relevant information in each modality and only share what is necessary. We find that such a strategy improves fusion performance, at the same time reducing computational cost. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens and VGGSound. All code and models will be released.


Introduction
Simultaneous multimodal sensations are a crucial enabler of human perceptual learning [57]. For artificial learning systems, however, designing a unified model for modality fusion is challenging due to a number of factors: (i) variations in learning dynamics between modalities [63], (ii) different noise topologies, with some modality streams containing more information for the task at hand than others, as well as (iii) specialised input representations. The difference in input representations between audio and vision is particularly stark -many state of the art audio classification methods rely on short term Fourier analysis to produce log-mel spectrograms, often using them as inputs to CNN architectures designed for images [29,55]. These time-frequency representations have different distributions to images -multiple acoustic objects can have energy at the same frequency, and the translation invariances of CNNs may no longer be a desired property (while an acoustic object can be shifted in time, a shift in frequency could alter the meaning entirely). In contrast, the visual stream in a video is three-dimensional (two spatial and one temporal), and while different spatial regions of the image correspond to different objects, there is the unique challenge of high redundancy across multiple frames. Hence input representations, and consequently neural network architectures and benchmarks tend to vary wildly for different modalities. For simplicity, the dominant paradigm for multimodal fusion therefore often consists of an ad-hoc scheme that involves integrating separate audio and visual networks via their output representations or scores i.e. 'late-fusion' [25,49].
In this work, we present a new transformer based model for audiovisual fusion in video. Despite originally being proposed for NLP tasks, there has been recent interest in transformers [61] as universal perceptual models [32], due to their ability to model dense correlations between tokens, at the same time making few assumptions about their inputs (and because continuous perceptual inputs can be tokenised). By dividing dense continuous signals into patches and rasterising them 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Figure 1: Cross-modal Fusion. Unlike late fusion (left), where no cross-modal information is exchanged in the model until after the classifier, we investigate two pathways for the exchange of cross-modal information. The first is via standard pairwise self attention across all hidden units in a layer, but applied only to later layers in the model -mid fusion (middle, left). We also propose the use of 'fusion bottlenecks' (middle, right) that restrict attention flow within a layer through tight latent units. Both forms of restriction can be applied in conjunction (Bottleneck Mid Fusion) for optimal performance (right). We show B = 2 bottleneck units and 3 hidden units per modality. Grey boxes indicate tokens that receive attention flow from both audio and video tokens.
to 1D tokens, transformers have been shown to perform competitively for image (ViT [18]) and video classification (ViViT [6]), and more recently, audio classification (AST [26]). Because these models are able to elegantly handle variable length sequences, a natural first extension would be to feed in a sequence of both visual and auditory patches to a transformer, with minimal changes to the architecture. This 'early fusion' model allows free attention flow between different spatial and temporal regions in the image, as well as across frequency and time in the audio spectrogram. While theoretically appealing, we hypothesise that full pairwise attention at all layers of the model is not necessary because audio and visual inputs contain dense, fine-grained information, much of which is redundant. This is particularly the case for video, as shown by the performance of 'factorised' versions of [6]. Such a model would also not scale well to longer videos due to the quadratic complexity of pairwise attention with token sequence length. To mitigate this, we propose two methods to restrict the flow of attention in our model. The first follows from a common paradigm in multimodal learning, which is to restrict cross-modal flow to later layers of the network, allowing early layers to specialise in learning and extracting unimodal patterns. Henceforth this is is referred to as 'mid fusion' (Fig. 1, middle left), where the layer at which cross-modal interactions are introduced is called the 'fusion layer'. The two extreme versions of this are 'early fusion' (all layers are cross-modal) and 'late fusion' (all are unimodal) which we compare to as a baselines. Our second idea (and main contribution), is to restrict cross-modal attention flow between tokens within a layer. We do this by allowing free attention flow within a modality, but force our model to collate and 'condense' information from each modality before sharing it with the other. The core idea is to introduce a small set of latent fusion units that form an 'attention bottleneck', through which cross-modal interactions within a layer must pass. We demonstrate that this 'bottlenecked' version, which we name Multimodal Bottleneck Transformer (MBT), outperforms or matches its unrestricted counterpart, but with lower computational cost.
Concretely, we make the following contributions: (i) We propose a new architecture (MBT) for audiovisual fusion. Our model restricts the flow of cross-modal information between latent units through tight fusion 'bottlenecks', that force the model to collect and 'condense' the most relevant inputs in each modality (and therefore share only that which is necessary with the other modality). This avoids the quadratic scaling cost of full pairwise attention, and leads to performance gains with less compute; (ii) We apply MBT to image and spectogram patches (Fig. 2), and explore a number of ablations related to the fusion layer, the sampling of inputs and data size; and finally (iii) We set the new state-of-the-art for video classification across a number of popular audio-visual benchmarks, including AudioSet [24], Epic-Kitchens100 [14] and VGGSound [12]. On the Audioset dataset, we outperform the current state of the art by 5.9 mAP (12.7% relative improvement).

Related work
Audiovisual learning: Audiovisual multimodal learning has a rich history, both before and during the deep learning era [53]. Given the limited available data and computational resources, early work focused on relatively simple early-stage (e.g. stacking hand-designed features) and late-stage  Figure 2: A Multimodal Fusion Transformer applied to audiovisual inputs. The input sequence consists of image and spectrogram patches. These are then projected into tokens and appended to special CLS (classification) and FSN (fusion bottleneck) tokens. Our transformer encoder then uses self attention to model unimodal information, and restricts cross-modal information flow via cross attention with the bottleneck tokens at multiple layers of the network.
(e.g. score fusion) techniques [13]. Deep learning has allowed more sophisticated strategies in which modality-specific or joint latents are implicitly learned to mediate the fusion. The result has enabled major advances in a range of downstream supervised audiovisual tasks [48,38,19]. In the supervised setting, multiple modality-specific convolution networks can be jointly trained, whose intermediate activations are then combined by summation [36] or via 'lateral connections' [64]. In the unsupervised setting, audiovisual learning is commonly used to learn good unimodal representations, with a popular pretraining task being to synchronise signals from different modalities via a contrastive loss [4,5,7,49,33,2,3], however each modality is usually encoded separately under this setup.
Multimodal transformers: The self attention operation of transformers provides a natural mechanism to connect multimodal signals. Multimodal transformers have been applied to various tasks including audio enhancement [19,60], speech recognition [27], image segmentation [66,60], crossmodal sequence generation [43,41,56], image and video retrieval [28,23,8], visual navigation [51] and image/video captioning/classification [46,59,58,40,31]. For many works, the inputs to transformers are the output representations of single modality CNNs [39,23] -unlike these works we use transformer blocks throughout, using only a single convolutional layer to rasterise 2D patches. The tokens from different modalities are usually combined directly as inputs to the transformers [42], for example, the recently released Perceiver model [32] introduces an iterative attention mechanism which takes concatenated raw multimodal signals as inputs, which corresponds to our 'early fusion' baseline. In contrast, we carefully examine the impact of different modality fusion strategies, including limiting cross-modal attention flow to later layers of our model, and 'channeling' cross-modal connections through bottlenecks in our proposed Multimodal Bottleneck Transformer (MBT).

Multimodal fusion transformers
In this section we describe our proposed Multimodal Bottleneck Transformer (MBT). We begin by summarising the recently proposed Vision Transformer (ViT) [18] and Audio Spectrogram Transformer (AST) [26], developed for image and audio classification respectively, in Sec. 3.1. We then describe our extension to the audio-visual fusion case. We discuss three different token fusion strategies (Sec. 3.2), and finally discuss the fusion pathway in the entire model (Sec. 3.3), which involves restricting multimodal fusion to certain layers of the model.

The ViT and AST architectures
Vision Transformer (ViT) [18] (and a recent extension to audio -Audio Spectrogram Transformer (AST) [26]) adapts the Transformer architecture [61], originally designed for natural language processing, to process 2D inputs with minimal changes. The key insight is to extract N nonoverlapping patches from the RGB image (or the audio spectrogram), x i ∈ R h×w , and convert them into a series of 1D tokens z i ∈ R d , as follows: Here, E is a linear projection mapping each token to R d , z cls is a special token prepended to this sequence so that its representation at the final layer can be passed to a classifier for classification tasks [17], and p ∈ R (N +1)×d is a learned positional embedding added to the tokens to retain positional information (as all subsequent self-attention operations are permutation invariant).
The tokens are then passed through an encoder consisting of a sequence of L transformer layers. Each transformer layer consists of Multi-Headed Self-Attention (MSA), Layer Normalisation (LN) and Multilayer Perceptron (MLP) blocks applied using residual connections. We denote a transformer layer, z l+1 = Transformer(z l ) as z l+1 = MLP(LN(y l )) + y l .
(3) Here, the MSA operation [61] computes dot-product attention [61] where the queries, keys and values are all linear projections of the same tensor, MSA(X) = Attention(W Q X, W K X, W V X). We further define Multi-Headed Cross Attention (MCA) between two tensors, X and Y, where X forms the query and Y forms the keys and values which are used to reweight the query as MCA(X, Y) = Attention(W Q X, W K Y, W V Y). This will be used in our multimodal case, as described next.

Multimodal transformer
We now describe our extension to the multimodal case. We begin by discussing three different token fusion strategies.

Fusion via vanilla self-attention
We begin by describing a 'vanilla' fusion model, which simply consists of the regular transformer applied to multimodal inputs. Our method of tokenising video is straightforward -given a video clip of length t seconds, we uniformly sample F RGB frames and convert the audio waveform into a single spectrogram. We then embed each frame and the spectrogram independently following the encoding proposed in ViT [18], and concatenate all tokens together into a single sequence.
Formally, if we have extracted a total of N v RGB patches from all F sampled frames, x rgb ∈ R Nv×d , and N a spectrogram patches, x spec ∈ R Na×d , our sequence of tokens is z = [z rgb ||z spec ] where z rgb = g(x rgb ; E rgb , z cls-rgb ) and z spec = g(x spec ; E spec , z cls-spec ). (4) Here, [z rgb ||z spec ] denotes the concatenation of the tokens for each modality. We use different projections E rgb and E spec for RGB and spectrogram patches respectively, and prepend a separate classification token for each modality.
Our multimodal encoder then applies a series of transformer layers in the same manner as above.
Attention is allowed to flow freely through the network, i.e. each RGB token can attend to all other RGB and spectrogram tokens as follows: z l+1 = Transformer(z l ; θ) with model parameters θ. Here Transformer refers to a standard transformer layer with vanilla self-attention blocks.

Fusion with modality-specific parameters
We can generalise this model by allowing each modality to have its own dedicated parameters θ rgb and θ spec , but still exchange information via the attention mechanism. For this purpose, we define a Cross-Transformer layer: z l+1 spec = Cross-Transformer(z l spec , z l ; θ spec ), where the Cross-Transformer employs the generalised cross-attention operation that takes two sets of inputs z 1 and z 2 that are not necessarily overlapping. This layer follows the original transformer layer with the difference being that Eq. 2 becomes Finally, note that we have explicitly defined the parameters, θ rgb and θ spec of the cross-transformer layers in Eq. 5 as they are different for each modality. However, when θ rgb and θ spec are equal, (θ rgb = θ spec = θ), the computation defined in Eq. 5 is equivalent to Sec. 3.2.1.

Fusion via attention bottlenecks
In order to tame the quadratic complexity of pairwise attention, we next introduce a small set of B fusion bottleneck tokens z fsn = [z 1 fsn , z 2 fsn , . . . , z B fsn ] to our input sequence (see Fig. 2). The input sequence is now z = [z rgb ||z fsn ||z spec ].
We then restrict all cross-modal attention flow in our model to be via these bottleneck tokens. More formally for layer l, we compute token representations as follows: Here i indexes each modality, in this case RGB and Spec, and z rgb and z spec can only exchange information via the bottleneck z fsn within a transformer layer. We first create modality specific temporary bottleneck fusion tokensẑ fsni , which are updated separately and simultaneously with audio and visual information (Equation 8). The final fusion tokens from each cross-modal update are then averaged in Equation 9. We also experiment with asymmetric updates for the bottleneck tokens (see appendix) and found performance was robust to this choice. We keep the number of bottleneck tokens in the network to be much smaller than the total number of latent units per modality (B N v and B N a ). Because all cross-modal attention flow must pass through these units, these tight 'fusion' bottlenecks force the model to condense information from each modality and share that which is necessary. As we show in the experiments, this increases or maintains performance for multimodal fusion, at the same time reducing computational complexity. We also note that our formulation is generic to the type and the number of modalities.

Where to fuse: early, mid and late
The above strategies discuss fusion within a layer, and in most transformer architectures (such as ViT), every layer consists of an identical set of operations. A common paradigm in multimodal learning, however, is to restrict early layers of a network to focus on unimodal processing, and only introduce cross-modal connections at later layers. This is conceptually intuitive if we believe lower layers are involved in processing low level features, while higher layers are focused on learning semantic concepts -low-level visual features such as edges and corners in images might not have a particular sound signature, and therefore might not benefit from early fusion with audio [64].
This can be implemented with our model as follows: We initially perform vanilla self-attention among tokens from a single modality for L f layers. Thereafter, we concatenate all latent tokens together, and pass them through the remaining L − L f layers where the tokens are fused according to Sec. 3.2. Here, L f = 0 corresponds to an 'early-fusion' model, L f = L a 'late-fusion' model, and 0 < L f < L a 'mid-fusion' one. More formally, this can be denoted as where Multimodal-Transformer(·) can refer to either of the 3 fusion strategies described in Sec 3.2.

Classification
For all model variants described above, we pass output representations of the CLS tokens z L cls-rgb and z L cls-spec to the same linear classifier and average the pre-softmax logits.

Experiments
We apply MBT to the task of video classification. In this section we first describe the datasets used to train and test multimodal fusion and their respective evaluation protocols (Sec. 4.1), then discuss implementation details (Sec. 4.2). We then ablate the key design choices in our model (Sec. 4.3), before finally comparing our model to the state of the art (Sec. 4.4).

Datasets and evaluation protocol
We experiment with three video classification datasets -AudioSet [24], Epic-Kitchens-100 [14] and VGGSound [12], described in more detail below. Results on two additional datasets Moments in Time [47] and Kinetics [35] are provided in the appendix. AudioSet [24] consists of almost 2 million 10-second video clips from YouTube, annotated with 527 classes. Like other YouTube datasets, this is a dynamic dataset (we only use the clips still available online). This gives us 20,361 clips for the balanced train set (henceforth referred to as mini-AudioSet or miniAS) and 18,589 clips for the test set. This test set is exactly the same as recent works we compare to, including Perceiver [32]. Instead of using the 2M unbalanced training set, we train on a (slightly more) balanced subset consisting of 500K samples (AS-500K). Details are provided in the appendix. Because each sample has multiple labels, we train with a binary cross-entropy (BCE) loss and report mean average precision (mAP) over all classes, following standard practice.
Epic-Kitchens 100 [14] consists of egocentric videos capturing daily kitchen activities. The dataset consists of 90,000 variable length clips spanning 100 hours. We report results for action recognition following standard protocol [14] -each action label is a combination of a verb and noun, and we predict both using a single network with two 'heads', both trained with a cross-entropy loss. The top scoring verb and action pair predicted by the network are used, and Top-1 action accuracy is the primary metric. Actions are mainly short-term (average length is 2.6s with minimum length 0.25s).
VGGSound [12] contains almost 200K video clips of length 10s, annotated with 309 sound classes consisting of human actions, sound-emitting objects and human-object interactions. Unlike AudioSet, the sound source for each clip is 'visually present' in the video. This was ensured during dataset creation through the use of image classifiers. After filtering clips that are no longer available on YouTube, we end up with 172,427 training and 14,448 test clips. We train with a standard crossentropy loss for classification and report Top-1 and Top-5 classification accuracy.

Implementation details
Our backbone architecture follows that of ViT [18] identically, specifically we use ViT-Base (ViT-B, L = 12, N H = 12, d = 3072) 1 initialised from ImageNet-21K [16], however we note that our method is agnostic to transformer backbone. Unless otherwise specialised, we use B = 4 bottleneck tokens for all experiments with bottleneck fusion. Bottleneck tokens are initialized using a Gaussian with mean of 0 and standard deviation of 0.02, similar to the positional embeddings in the public ViT [18] code. We randomly sample clips of t seconds for training. RGB frames for all datasets are extracted at 25 fps. For AudioSet and VGGSound we sample 8 RGB frames over the sampling window of length t with a uniform stride of length (t × 25)/8. We extract 16 × 16 patches from each frame of size 224 × 224, giving us a total of 8 × 14 × 14 = 1568 patches per video. For Epic-Kitchens (because the segments are shorter), we sample 32 frames with stride 1. Audio for all datasets is sampled at 16kHz and converted to mono channel. Similar to [26], we extract log mel spectrograms with a frequency dimension of 128 computed using a 25ms Hamming window with hop length 10ms. This gives us an input of size 128 × 100t for t seconds of audio. Spectrogram patches are extracted with size 16 × 16, giving us 50 × 8 = 400 patches for 8 seconds of audio. For images we apply the standard data augmentations used in [6] (random crop, flip, colour jitter), and for spectrograms we use SpecAugment [50] with a max time mask length of 192 frames and max frequency mask length of 48 bins following AST [26]. We set the base learning rate to 0.5 and train for 50 epochs, using Mixup [67] with α = 0.3 and stochastic depth regularisation [30] with probability p = 0.3. All models (across datasets) are trained with a batch size of 64, synchronous SGD with momentum of 0.9, and a cosine learning rate schedule with warmup of 2.5 epochs on TPU accelerators using the Scenic library [15]. Inference: Following standard practice, we uniformly sample multiple temporal crops from the clip and average per-view logits to obtain the final result. The number of test crops is set to 4.

Ablation analysis
In this section we investigate the impact of the different architectural choices in MBT. Unless otherwise specified, we use the mini-AudioSet split for training and report results on the AudioSet eval split. More ablations on backbone size and pretraining initalisation can be found in the appendix.

Fusion strategies
We implement all the three fusion strategies described in Sec. 3.2: (i) Vanilla self-attention -Unrestricted pairwise attention between all latent units within a layer; (ii) Vanilla cross-attention with separate weights: Same as above, but we now have separate weights for each modality. The latent units are updated via pairwise attention with all other latent units from both modalities; and finally (iii) Bottleneck fusion: Here all cross-modal attention must pass through bottleneck fusion latents. Note that these three fusion strategies only describe attention flow between tokens within a layer. For strategies (ii) and (iii), we also conduct experiments showing the impact of restricting cross-modal attention to layers after a fixed fusion layer L f . We investigate models with different fusion layers, L f = 0, 2, 4, 6, 8, 10, 12, and present the results in Fig. 3. 2 Sharing weights for both modalities: We first investigate the impact of sharing the encoder weights for both modalities (strategy (i) vs (ii)). The results can be found in Fig. 7 in the appendix. When modalities are fused at earlier layers, using separate encoders improves performance. For models with later fusion layers, performance is similar for both models. We hence use separate modality weights for further experiments. We fix the input span t to 4s and the number of bottleneck tokens B to 4. We conduct 3 runs for each experiment and report mean and std deviation. As can be seen from Fig. 3 (left), 'mid fusion' outperforms both early (L f = 0) and late fusion (L f = 12), with optimal performance obtained by using fusion layer L f = 10 for vanilla cross-attention and L f = 8 for bottleneck attention. This suggests that the model benefits from restricting cross-modal connections to later layers, allowing earlier layers to specialise to learning unimodal features, however still benefits from multiple layers of cross-modal information flow. In appendix D , we confirm that mid fusion outperforms late fusion across a number of different datasets. Attention bottlenecks: In Fig. 3, we also examine the effect of bottleneck attention vs vanilla cross-attention for multimodal fusion. We find that for all values of L f restricting flow to bottlenecks improves or maintains performance, with improvements more prominent at lower values of L f . At L f = 10, both perform similarly, note that at this stage we only have 3 fusion layers in the model. Our best performing model uses attention bottlenecks with L f = 8, and we fix this for all further experiments. We also compare the amount of computation, measured in GFLOPs, for both fusion strategies (Fig. 3, right). Using a small number of bottleneck tokens (in our experiments B = 4) adds negligible extra computation over a late fusion model, with computation remaining largely constant with varying fusion layer L f . This is in contrast to vanilla cross-fusion, which has a non-negligible computational cost for every layer it is applied to. We note that for early fusion (L f = 0), bottleneck fusion outperforms vanilla cross-attention by over 2 mAP, with less than half the computational cost.

Number of bottleneck tokens B:
We experiment with B = 4, 36, 64, 256 and 1024, and find that performance is relatively consistent (all within 0.5 mAP). We hence fix the number of tokens to B = 4 for all experiments. It is interesting that with such a small number of cross-modal connections through only 4 hidden units (B = 4) at each cross-modal layer, we get large performance gains over late fusion (Fig. 3), highlighting the importance of allowing cross-modal information to flow at multiple layers of the model.

Input sampling and dataset size
In this section we investigate the impact of different modality sampling strategies. We also compare to single modality baselines -the visual-only and audio-only baselines consist of a vanilla transformer model applied to only the RGB or spectrogram patches respectively. Sampling window size t: An advantage of our transformer based model is that we can easily input variable length token sequences. We experiment with varying the sampling window t with the following values t = 2, 4, 6 and 8 seconds (note that all videos in AudioSet are 10s), and show results in Fig. 4 3 . At inference, we uniformly sample multiple windows covering the entire video. While the number of spectrogram patches N a changes with t, we keep the number of RGB patches N v     fixed by changing the stride of frames (to avoid running out of memory). Our results indicate that the performance of both the audio and audio-visual fusion model increases with input span, however the performance of the visual-only model slightly decreases (we hypothesize that this is due to the increased fixed stride, meaning fewer frames are randomly sampled during training). We fix t = 8s in all further experiments. Synchronous vs asynchronous sampling: Given that auditory and visual events may not always be perfected aligned in videos [36], we also investigate asynchronous sampling of different modalities.
Here input windows are sampled independently from the entire video clip for each modality. Results are provided in Fig. 8 in the appendix. We find performance to be largely robust to either case, and so for simplicity we use synchronised sampling for all further experiments. Modality MixUp: While applying Mixup regularization [67] to training, we note that there are two different ways to apply it for multimodal inputs -the standard approach is to sample one set of mixup weights from a Beta distribution using the parameter α, and use it to generate all virtual modality-label pairs [67]. We also explore a modified version which we call modality mixup, which samples an independent weight for each modality. Modality mixup imposes stronger augmentation than standard mixup, leading to a slight improvement (42.6 mAP to 43.9 mAP) on AudioSet. Impact of dataset size: We show the impact of varying the number of training samples in Fig. 5, and find a monotonic increase with dataset size (more steeply for audio-only than visual-only).

Results
Comparison to single modality performance: We compare MBT to visual-only and audio-only baselines on AudioSet (Table 1), Epic-Kitchens (Table 2) and VGGSound (Table 3). Note we use the best parameters obtained via the ablations above, i.e. bottleneck fusion with t = 8, B = 4, F l = 8 and modality mixup. For all datasets, multimodal fusion outperforms the higher-performing single modality baseline, demonstrating the value of complementary information. The relative importance of modalities for the classification labels varies (audio-only has higher relative performance for AudioSet and lower for Epic-Kitchens, while both audio and visual baselines are equally strong for VGGSound). This is (unsurprisingly) largely a function of the dataset annotation procedure and positions VGGSound as a uniquely suitable dataset for fusion. We also show that audio-visual fusion    a relatively new dataset, we compare to two existing audio-only works 4 (Table 3), and set the first audiovisual benchmark (that we are aware of) on this dataset. Visualisation of attention maps Finally, we compute maps of the attention from the output CLS tokens to the RGB image input space using Attention Rollout [1]. Results on test images for both a vanilla fusion model and MBT trained on Audioset-mini (fusion layer L f = 8) are shown in Figure 6. We show the attention maps summed over all the frames in the video clip. We note that first, the model focuses on semantically salient regions in the video for audio classification, particularly regions where there is motion that creates or modifies sound, i.e. the mouth of humans making sounds, fingertips on a piano, hands and instruments. This is unlike state of the art sound source localisation techniques trained with images [11], which tend to highlight the entire object. We further note that the attention maps for MBT are more localised to these regions, showing that the tight bottlenecks do force the model to focus only on the image patches that are actually relevant for the audio classification task and which benefit from early fusion with audio.

Conclusion
We propose a new transformer architecture (MBT) for audiovisual fusion, and explore a number of different fusion strategies using cross-attention between latent tokens. We propose a novel strategy to restrict cross-modal attention via a small set of fusion 'bottlenecks', and demonstrate that this improves performance over vanilla cross-attention at lower computational cost, achieving state of the art results on a number of benchmarks. Future work will involve extending MBT to other modalities such as text and optical flow. Limitations: The fusion layer is a hyperparameter and may need to be tuned specifically for different tasks and datasets. We also only explore fully supervised fusion, and future work will tackle extensions to a self-supervised learning framework. Broader impact: Multimodal fusion strategies are important for machine learning, as fusing complementary information from different modalities can increase robustness when applied to real world applications. We also note that transformers are in general compute-heavy, which can have adverse environmental effects. We propose a token fusion method via bottlenecks that helps reduce computational complexity when applying transformers for multimodal fusion. Finally, we observe that training datasets contain biases that may render models trained on them unsuitable for certain applications. It is thus possible that people use classification models (intentionally or not) to make decisions that impact different groups in society differently, and it is important to keep this in mind when deploying, analysing and building upon these models.
Here the bottleneck tokens are updated twice, first with visual information (Equation 10), and then with audio information (Equation 11). We also experimented with updating the bottlenecks with audio information first and compare both variations to the symmetric update in Table 4. We find performance is robust to all variations.  Table 4: Asymmetric vs symmetric bottleneck updates.

A.2 Backbone architecture
We experiment with three standard ViT [18] backbones, ViT-Small, ViT-Base and ViT-Large on both Audioset-mini and VGGSound. We report results in Table 5 for audiovisual fusion with our best MBT model. We find that performance increases from ViT-Small to ViT-Base, but then drops for ViT-Large. This could be due to the fact that these datasets are on the smaller side, and more data might be required to take advantage of larger models.
Backbone AS-mini VGGSound  Table 5: Performance with varying backbones on AS-mini and VGGSound.

A.3 The impact of weight sharing
We investigate the impact of sharing the encoder weights for both modalities (strategy (i) vs (ii)) as described in Sec. 4.3.1 . Results are provided in Fig. 7 for different fusion layers L f . When modalities are fused at earlier layers, using separate encoders improves performance. For models with later fusion layers, performance is similar for both models.

A.4 Input sampling
Here we investigate asynchronous sampling of different modalities (where input windows are sampled independently from the entire video clip for each modality) as compared to synchronous sampling. Results are provided in Fig. 8 for different input span lengths t. Over multiple runs we find that performance is largely robust to either sampling choice. We hypothesise that asynchronous sampling provides the following trade-off: while it introduces a misalignment between the two modality inputs, slight shifts are also a good source of temporal augmentation. As the video clip span length grows, the possible options for misalignment between inputs are less severe, while the impact of additional augmentation is more evident.
In Table 6, we provide the results in numerical form used to create Fig. 4 . We perform 3 runs per experiment and report mean and standard deviation. All segments in AudioSet are 10 seconds long.

B Per class performance
We also examine per-class average precision (AP) results for our best model trained on the mini-Audioset (note that this dataset has 527 classes). We first show the results for the 60 top ranked classes in Audioset (by audio-visual mAP performance) in Fig. 9. We show the per class AP using our best fusion model (MBT), as well as the performance of audio only and visual only baselines. Audio-visual fusion improves performance over audio only or visual only for almost all (57 out of 60) classes, except for 'bagpiping', 'emergency vehicle' and 'didgeridoo' which have strong audio signatures. We then analyse the top 60 classes for which fusion has the largest improvement over single modality performance, over audio-only (Figure 10, top) and visual-only (Figure 10, bottom). For some classes such as 'bicycle' and 'shuffling cards', fusion improves over the audio-only baseline by over 60% in absolute AP. The class that benefits most from audio-visual fusion over a visual-only baseline is 'Whistling' (almost 80% improvement in absolute AP).

C Additional Datasets
In this section we report results on 2 additional datasets, Moments in Time [47] and Kinetics [35].

C.1 Moments In Time
Moments In Time [47] consists of 800,000, 3-second clips from YouTube videos. The videos are diverse and capture dynamic scenes involving animals, objects, people, or natural phenomena. The videos are labelled with 330 verb classes, each associated with over 1,000 videos. We show results for MBT compared to single modality baselines in Table 7. Our first observation is that audio-only performance is much lower than visual-only. This is largely a function of the annotation procedure for the dataset, however we also note that clips are only 3 seconds long, and as shown in Fig. 4 , audio-only performance is heavily dependant on the span length t on Audioset, suggesting that it may be difficult to recognise audio events from shorter inputs. Our fusion model provides a further modest 1% boost to performance over the visual-only baseline.

C.2 Kinetics
Kinetics [35] consists of 10-second videos sampled at 25fps from YouTube. We evaluate on both Kinetics 400 [35] and a commonly used subset Kinetics-Sound [4], containing 400 and 36 classes respectively. As these are dynamic datasets (videos may be removed from YouTube), we train and test on 209,552 and 17,069 videos respectively for Kinetics and report results on 1,165 videos for Kinetics-Sound. Results for MBT compared to single modality baselines are shown in Table 8. We note that on the entire Kinetics test set, our fusion model outperforms the visual only baseline by about 1% in top 1 accuracy (in line with other works [64] that demonstrate that audio for the large part does not improve performance for most Kinetics classes). This gap is widened, however, for the

AP
Difference between fusion and visual-only performance Figure 10: Top 60 classes that have the highest gain with fusion over a audio only (top) and visual only (bottom) baseline. Note how fusion improves the per class AP for certain classes by over 50% over a unimodal model. As expected, the classes that benefit most from visual information are 'bicycle' and 'shuffling cards' and the class that benefits most from audio is 'Whistling'.
Kinetics-Sound subset of the dataset (over 4%), as expected because this subset consists of classes in Kinetics selected to have a strong audio signature [4].

D Dataset Variations for MBT vs Late Fusion
In this section we further analyse the significance of our method across all the popular video classification datasets used in the paper (most ablations results are only shown for mini-Audioset in the main paper). We note that the gap between MBT and late-fusion is highly dataset dependant (see Table 9), with our method providing an even greater advantage for Epic-Kitchens (almost 6% difference in Top 1 action accuracy).   Table 8: Comparison to state of the art on Kinetics [35] and Kinetics Sound [4]. We report top-1 and top-5 classification accuracy. AV: Refers to audio-visual fusion. † Note the Kinetics-Sound test set has reduced since this work as videos have been removed from YouTube, hence this is not a direct comparison.

E Transfer learning
We use checkpoints pretrained on VGGSound, Kinetics400 and AS-500K and finetune them on Audioset-mini and VGGSound (note we use a ViT-B backbone for these experiments, and report results for audiovisual fusion with our best MBT model). Results are provided in Table 10. While Kinetics400 pretraining gives a slight 0.7% mAP boost on AS-mini, VGGSound initialisation gives a substantial 3% mAP boost over Imagenet Initialisation. On VGGSound, AS500K pretraining gives a more modest boost of 1.2% Top 1 Acc, while Kinetics pretraining does not help (expected as VGGSound is a larger dataset).

F AS-500K details
The original unbalanced AudioSet training set consists of almost 2M samples, and is extremely unbalanced with most samples either labelled as speech or music. To improve training efficiency, we create a slightly more balanced subset called AudioSet-500K. The main issue is that AudioSet is multilabel, and this makes balancing difficult. We create AS-500K by greedily restricting the maximum number of samples per class to be 200K. Given the distribution of labels, this gives us a total size of 508,994 samples. We provide the full histogram of labels in Fig. 11 (note the number of samples is on a log 10 scale).    Log (# samples) Figure 11: Class label histogram in the AudioSet-500K split.