{"title": "Modulating early visual processing by language", "book": "Advances in Neural Information Processing Systems", "page_first": 6594, "page_last": 6604, "abstract": "It is commonly assumed that language refers to high-level visual concepts while leaving low-level visual processing unaffected. This view dominates the current literature in computational models for language-vision tasks, where visual and linguistic inputs are mostly processed independently before being fused into a single representation. In this paper, we deviate from this classic pipeline and propose to modulate the \\emph{entire visual processing} by a linguistic input. Specifically, we introduce Conditional Batch Normalization (CBN) as an efficient mechanism to modulate convolutional feature maps by a linguistic embedding. We apply CBN to a pre-trained Residual Network (ResNet), leading to the MODulatEd ResNet (\\MRN) architecture, and show that this significantly improves strong baselines on two visual question answering tasks. Our ablation study confirms that modulating from the early stages of the visual processing is beneficial.", "full_text": "Modulating early visual processing by language\n\nHarm de Vries\u2217\nUniversity of Montreal\n\nmail@harmdevries.com\n\nFlorian Strub\u2217\n\nUniv. Lille, CNRS, Centrale Lille,\n\nJ\u00e9r\u00e9mie Mary\u2020\n\nUniv. Lille, CNRS, Centrale Lille,\n\nInria, UMR 9189 CRIStAL\nflorian.strub@inria.fr\n\nInria, UMR 9189 CRIStAL\n\njeremie.mary@univ-lille3.fr\n\nHugo Larochelle\n\nGoogle Brain\n\nOlivier Pietquin\n\nDeepMind\n\nAaron Courville\n\nUniversity of Montreal\n\nhugolarochelle@google.com\n\npietquin@google.com\n\naaron.courville@gmail.com\n\nAbstract\n\nIt is commonly assumed that language refers to high-level visual concepts while\nleaving low-level visual processing unaffected. This view dominates the current\nliterature in computational models for language-vision tasks, where visual and\nlinguistic inputs are mostly processed independently before being fused into a\nsingle representation. In this paper, we deviate from this classic pipeline and\npropose to modulate the entire visual processing by a linguistic input. Speci\ufb01cally,\nwe introduce Conditional Batch Normalization (CBN) as an ef\ufb01cient mechanism\nto modulate convolutional feature maps by a linguistic embedding. We apply\nCBN to a pre-trained Residual Network (ResNet), leading to the MODulatEd\nResNet (MODERN) architecture, and show that this signi\ufb01cantly improves strong\nbaselines on two visual question answering tasks. Our ablation study con\ufb01rms that\nmodulating from the early stages of the visual processing is bene\ufb01cial.\n\n1\n\nIntroduction\n\nHuman beings combine the processing of language and vision with apparent ease. For example, we\ncan use natural language to describe perceived objects and we are able to imagine a visual scene\nfrom a given textual description. Developing intelligent machines with such impressive capabilities\nremains a long-standing research challenge with many practical applications.\nTowards this grand goal, we have witnessed an increased interest in tasks at the intersection of\ncomputer vision and natural language processing.\nIn particular, image captioning [16], visual\nquestion answering (VQA)[1, 23] and visually grounded dialogue systems[5, 6] constitute a popular\nset of example tasks for which large-scale datasets are now available. Developing computational\nmodels for language-vision tasks is challenging, especially because of the open question underlying\nall these tasks: how to fuse/integrate visual and textual representations? To what extent should we\nprocess visual and linguistic input separately, and at which stage should we fuse them? And equally\nimportant, what fusion mechanism to use?\nIn this paper, we restrict our attention to the domain of visual question answering which is a natural\ntestbed for fusing language and vision. The VQA task concerns answering open-ended questions\nabout images and has received signi\ufb01cant attention from the research community [1, 9, 17, 23].\nCurrent state-of-the-art systems often use the following computational pipeline [2, 17, 20] illustrated\nin Fig 1. They \ufb01rst extract high-level image features from an ImageNet pretrained convolutional\nnetwork (e.g. the activations from a ResNet network [12]), and obtain a language embedding using a\n\n\u2217The \ufb01rst two authors contributed equally\n\u2020Now at Criteo\n\n\fFigure 1: An overview of the classic VQA pipeline (left) vs ours (right). While language and vision\nmodalities are independently processed in the classic pipeline, we propose to directly modulate\nResNet processing by language.\n\nrecurrent neural network (RNN) over word-embeddings. These two high-level representations are\nthen fused by concatenation [17], element-wise product [11, 13, 14, 17], Tucker decomposition [2]\nor compact bilinear pooling [9], and further processed for the downstream task at hand. Attention\nmechanisms [27] are often used to have questions attend to speci\ufb01c spatial locations of the extracted\nhigher-level feature maps.\nThere are two main reasons for why the recent literature has focused on processing each modality\nindependently. First, using a pretrained convnet as feature extractor prevents over\ufb01tting; Despite a\nlarge training set of a few hundred thousand samples, backpropagating the error of the downstream\ntask into the weights of all layers often leads to over\ufb01tting. Second, the approach aligns with the\ndominant view that language interacts with high-level visual concepts. Words, in this view, can be\nthought of as \u201cpointers\u201d to high-level conceptual representations. To the best of our knowledge, this\nwork is the \ufb01rst to fuse modalities at the very early stages of the image processing.\nIn parallel, the neuroscience community has been exploring to what extent the processing of language\nand vision is coupled [8]. More and more evidence accumulates that words set visual priors which\nalter how visual information is processed from the very beginning [3, 15, 24]. More precisely, it is\nobserved that P1 signals, which are related to low-level visual features, are modulated while hearing\nspeci\ufb01c words [3]. The language cue that people hear ahead of an image activates visual predictions\nand speed up the image recognition process. These \ufb01ndings suggest that independently processing\nvisual and linguistic features might be suboptimal, and fusing them at the early stage may help the\nimage processing.\nIn this paper, we introduce a novel approach to have language modulate the entire visual processing of\na pre-trained convnet. We propose to condition the batch normalization [21] parameters on linguistic\ninput (e.g., a question in a VQA task). Our approach, called Conditional Batch Normalization (CBN),\nis inspired by recent work in style transfer [7]. The key bene\ufb01t of CBN is that it scales linearly with\nthe number of feature maps in a convnet, which impacts less than 1% of the parameters, greatly\nreducing the risk of over-\ufb01tting. We apply CBN to a pretrained Residual Network, leading to a\nnovel architecture to which we refer as MODERN. We show signi\ufb01cant improvements on two VQA\ndatasets, VQAv1 [1] and GuessWhat?! [6], but stress that our approach is a general fusing mechanism\nthat can be applied to other multi-modal tasks.\nTo summarize, our contributions are three fold:\n\u2022 We propose conditional batch normalization to modulate the entire visual processing by language\n\nfrom the early processing stages,\n\n\u2022 We condition the batch normalization parameters of a pretrained ResNet on linguistic input, leading\n\nto a new network architecture: MODERN,\n\n\u2022 We demonstrate improvements on state-of-the-art models for two VQA tasks and show the\n\ncontribution of this modulation on the early stages.\n\n2\n\n\f2 Background\n\nIn this section we provide preliminaries on several components of our proposed VQA model.\n\n2.1 Residual networks\n\nWe brie\ufb02y outline residual networks (ResNets) [12], one of the current top-performing convolutional\nnetworks that won the ILSVRC 2015 classi\ufb01cation competition. In contrast to precursor convnets\n(e.g. VGG[22]) that constructs a new representation at each layer, ResNet iteratively re\ufb01nes a\nrepresentation by adding residuals. This modi\ufb01cation enables to train very deep convolutional\nnetworks without suffering as much from the vanishing gradient problem. More speci\ufb01cally, ResNets\nare built from residual blocks:\n\nF k+1 = ReLU(F k + R(F k))\n\n(1)\nwhere F k denotes the outputted feature map. We will refer to Fi,c,w,h to denote the ith input\nsample of the cth feature map at location (w, h). The residual function R(F k) is composed of three\nconvolutional layers (with a kernel size of 1, 3 and 1, respectively). See Fig. 2 in the original ResNet\npaper [12] for a detailed overview of a residual block.\nA group of blocks is stacked to form a stage of computation in which the representation dimensionality\nstays identical. The general ResNet architecture starts with a single convolutional layer followed by\nfour stages of computation. The transition from one stage to another is achieved through a projection\nlayer that halves the spatial dimensions and doubles the number of feature maps. There are several\npretrained ResNets available, including ResNet-50, ResNet-101 and ResNet-152 that differ in the\nnumber of residual blocks per stage.\n\n2.2 Batch Normalization\n\nThe convolutional layers in ResNets make use of Batch Normalization (BN), a technique that was\noriginally designed to accelarate the training of neural networks by reducing the internal co-variate\nshift [21]. Given a mini-batch B = {Fi,\u00b7,\u00b7,\u00b7}N\ni=1 of N examples, BN normalizes the feature maps at\ntraining time as follows:\n\nBN (Fi,c,h,w|\u03b3c, \u03b2c) = \u03b3c\n\n+ \u03b2c,\n\n(2)\n\nFi,c,w,h \u2212 EB[F\u00b7,c,\u00b7,\u00b7]\n\n(cid:112)VarB[F\u00b7,c,\u00b7,\u00b7] + \u0001\n\nwhere \u0001 is a constant damping factor for numerical stability, and \u03b3c and \u03b2c are trainable scalars\nintroduced to keep the representational power of the original network. Note that for convolutional\nlayers the mean and variance are computed over both the batch and spatial dimensions (such that\neach location in the feature map is normalized in the same way). After the BN module, the output\nis fed to a non-linear activation function. At inference time, the batch mean EB and variance VarB\nare replaced by the population mean \u00b5 and variance \u03c32, often estimated by an exponential moving\naverage over batch mean and variance during training.\n\n2.3 Language embeddings\n\nWe brie\ufb02y recap the most common way to obtain a language embedding from a natural language\nquestion. Formally, a question q = [wk]K\nk=1 is a sequence of length K with each token wk taken\nfrom a prede\ufb01ned vocabulary V . We transform each token into a dense word-embedding e(wk)\nby a learned look-up table. For task with limited linguistic corpora (like VQA), it is common to\nconcatenate pretrained Glove[19] vectors to the word embeddings. The sequence of embeddings\nk=1 is then fed to a recurrent neural network (RNN), which produces a sequence of RNN\n[e(wk)]K\nstate vectors [sk]K\n\nk=1 by repeatedly applying the transition function f:\n\n(3)\nPopular transition functions, like a long-short term memory (LSTM) cell [10] and a Gated Recurrent\nUnit (GRU)[4], incorporate gating mechanisms to better handle long-term dependencies. In this work,\nwe will use an LSTM cell as our transition function. Finally, we take the last hidden state sI as the\nembedding of the question, which we denote as eq throughout the rest of this paper.\n\nsk+1 = f (sk, e(wk)).\n\n3\n\n\fFigure 2: An overview of the computation graph of batch normalization (left) and conditional batch\nnormalization (right). Best viewed in color.\n\n3 Modulated Residual Networks\n\nIn this section we introduce conditional batch normalization, and show how we can use it to modulate\na pretrained ResNet. The key idea is to predict the \u03b3 and \u03b2 of the batch normalization from a\nlanguage embedding. We \ufb01rst focus on a single convolutional layer with batch normalization module\nBN (Fi,c,h,w|\u03b3c, \u03b2c) for which pretrained scalars \u03b3c and \u03b2c are available. We would like to directly\npredict these af\ufb01ne scaling parameters from our language embedding eq. When starting the training\nprocedure, these parameters must be close to the pretrained values to recover the original ResNet\nmodel as a poor initialization could signi\ufb01cantly deteriorate performance. Unfortunately, it is dif\ufb01cult\nto initialize a network to output the pretrained \u03b3 and \u03b2. For these reasons, we propose to predict\na change \u2206\u03b2c and \u2206\u03b3c on the frozen original scalars, for which it is straightforward to initialize a\nneural network to produce an output with zero-mean and small variance.\nWe use a one-hidden-layer MLP to predict these deltas from the question embedding eq for all feature\nmaps within the layer:\n\n\u2206\u03b2 = M LP (eq)\n\n\u2206\u03b3 = M LP (eq)\n\n(4)\n\nSo, given a feature map with C channels, these MLPs output a vector of size C. We then add these\npredictions to the \u03b2 and \u03b3 parameters:\n\n\u02c6\u03b2c = \u03b2c + \u2206\u03b2c\n\n\u02c6\u03b3c = \u03b3c + \u2206\u03b3c\n\n(5)\n\nthese updated \u02c6\u03b2 and \u02c6\u03b3 are used as parameters for\n\nFinally,\nthe batch normalization:\nBN (Fi,c,h,w| \u02c6\u03b3c, \u02c6\u03b2c)). We stress that we freeze all ResNet parameters, including \u03b3 and \u03b2, dur-\ning training. In Fig. 2, we visualize the difference between the computational \ufb02ow of the original\nbatch normalization and our proposed modi\ufb01cation. As explained in section 2.1, a ResNet consists of\nfour stages of computation, each subdivided in several residual blocks. In each block, we apply CBN\nto the three convolutional layers, as highlighted in Fig. 3.\nCBN is a computationally ef\ufb01cient and powerful method to modulate neural activations; It enables\nthe linguistic embedding to manipulate entire feature maps by scaling them up or down, negating\nthem, or shutting them off, etc. As there only two parameters per feature map, the total number of\nBN parameters comprise less than 1% of the total number of parameters of a pre-trained ResNet.\nThis makes CBN a very scalable method compared to conditionally predicting the weight matrices\n(or a low-rank approximation to that).\n\n4 Experimental setting\n\nWe evaluate the proposed conditional batch normalization on two VQA tasks. In the next section, we\noutline these tasks and describe the neural architectures we use for our experiments. The source code\nfor our experiments is available at https://github.com/GuessWhatGame. The hyperparameters\nare also provided in Appendix A.\n\n4\n\n\fFigure 3: An overview of the MODERN architecture conditioned on the language embedding.\nMODERN modulates the batch norm parameters in all residual blocks.\n\n4.1 VQA\n\nThe Visual Question Answering (VQA) task consists of open-ended questions about real images. An-\nswering these questions requires an understanding of vision, language and commonsense knowledge.\nIn this paper, we focus on VQAv1 dataset [1], which contains 614K questions on 204K images.\nOur baseline architecture \ufb01rst obtains a question embedding eq by an LSTM-network, as further\ndetailed in section 2.3. For the image, we extract the feature maps F of the last layer of ResNet-50\n(before the pooling layer). For input of size 224x224 these feature maps are of size 7x7, and we\nincorporate a spatial attention mechanism, conditioned on the question embedding eq, to pool over\nthe spatial dimensions. Formally, given a feature maps Fi,\u00b7,\u00b7,\u00b7 and question embedding eq, we obtain\na visual embedding ev as follows:\n\n\u03bew,h = M LP ([Fi,\u00b7,w,h; eq])\n\n; \u03b1w,h =\n\n;\n\nev =\n\n\u03b1w,hFi,\u00b7,w,h\n\n(6)\n\n(cid:80)\n\nexp(\u03bew,h)\nw,h exp(\u03bew,h)\n\n(cid:88)\n\nw,h\n\nwhere [Fi,\u00b7,w,h; eq] denotes concatenating the two vectors. We use an MLP with one hidden layer\nand ReLU activations whose parameters are shared along the spatial dimensions. The visual and\nquestion embedding are then fused by an element-wise product [1, 13, 14] as follows:\n\nfuse(eq, ev) = P T(cid:0)(tanh(U T eq)) \u25e6 (tanh(V T ev))(cid:1) + bP ,\n\n(7)\nwhere \u25e6 denotes an element-wise product, and P , U and V are trainable weight matrices and bP\nis a trainable bias. The linguistic and perceptual representations are \ufb01rst projected to a space of\nequal dimensionality, after which a tanh non-linearity is applied. A fused vector is then computed by\nan element-wise product between the two representations. From this joined embedding we \ufb01nally\npredict an answer distribution by a linear layer followed by a softmax activation function.\nWe will use the described architecture to study the impact CBN when using it in several stages of\nthe ResNet. As our approach can be combined with any existing VQA architecture, we also apply\nMODERN to MRN [13, 14], a state-of-the-art network for VQA More speci\ufb01cally, this network\nreplaces the classic attention mechanism with a more advanced one that included g glimpses over the\nimage features:\n\n\u03beg\nw,h = P T\n\n(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)g\n\n\u03b1g (tanh(U(cid:48)T q) \u25e6 tanh(V (cid:48)T F T\n(cid:88)\n\n\u03b1g\nw,hFi,\u00b7,w,h\n\nev =\n\nw,h\n\ni,\u00b7,w,h)))\n\n; \u03b1g\n\nw,h =\n\n(cid:80)\n\nexp(\u03beg\nw,h)\nw,h exp(\u03beg\n\nw,h)\n\n(8)\n\n(9)\n\nwhere P\u03b1g is a trainable weight matrix de\ufb01ned for each glimpse g, U(cid:48) and V (cid:48) are trainable weight\nmatrices shared among the glimpses and (cid:107) concatenate vectors over their last dimension.\n\n5\n\n\fTable 1: VQA accuracies trained with train set and evaluated on test-dev.\n\nAnswer type\n\n4 Baseline\n\n2\n2\nx\n4\n2\n2\n\n8\n4\n4\nx\n8\n4\n4\n\nFt Stage 4\nFt BN\nMODERN\nMRN [14] with ResNet-50\nMRN [14] with ResNet-152\nMUTAN+MLB [2]\nMCB + Attention [9] with ResNet-50\nMCB + Attention [9] with ResNet-152\nMODERN\nMODERN + MRN [14]\n\nYes/No\n79.45%\n78.37%\n80.18%\n81.17%\n80.20%\n80.95%\n82.29%\n60.46%\n\n-\n\n81.38%\n82.17%\n\nNumber\n36.63%\n34.27%\n35.98%\n37.79%\n37.73%\n38.39%\n37.27%\n38.29%\n\n-\n\n36.06%\n38.06%\n\nOther\n44.62%\n43.72%\n46.07%\n48.66%\n49.53%\n50.59%\n48.23%\n48.68%\n\n-\n\n51.64%\n52.29%\n\nOverall\n58.05%\n56.91%\n58.98%\n60.82%\n60.84%\n61.73%\n61.02%\n60.46%\n62.50%\n62.16%\n63.01%\n\nTable 2: Ablation study to investigate the impact of leaving out the lower stages of ResNet.\n\n(a) VQA, higher is better\nCBN applied to Val. accuracy\n\u2205\nStage 4\nStages 3 \u2212 4\nStages 2 \u2212 4\nAll\n\n56.12%\n57.68%\n58.29%\n58.32%\n58.56%\n\n(b) GuessWhat?!, lower is better\n\nCBN applied to\n\u2205\nStage 4\nStages 3 \u2212 4\nStages 2 \u2212 4\nAll\n\nTest error\n29.92%\n26.42%\n25.24%\n25.31%\n25.06%\n\nNoticeably, MODERN modulates the entire visual processing pipeline and therefore backpropagates\nthrough all convolutional layers. This requires much more GPU memory than using extracted features.\nTo feasibly run such experiments on today\u2019s hardware, we conduct all experiments in this paper with\na ResNet-50.\nAs for our training procedure, we select the 2k most-common answers from the training set, and\nuse a cross-entropy loss over the distribution of provided answers. We train on the training set, do\nearly-stopping on the validation set, and report the accuracies on the test-dev using the evaluation\nscript provided by [1].\n\n4.2 GuessWhat?!\n\nGuessWhat?! is a cooperative two-player game in which both players see the image of a rich visual\nscene with several objects. One player \u2013 the Oracle \u2013 is randomly assigned an object in the scene.\nThis object is not known by the other player \u2013 the questioner \u2013 whose goal it is to locate the hidden\nobject by asking a series of yes-no questions which are answered by the Oracle [6].\nThe full dataset is composed of 822K binary question/answer pairs on 67K images. Interestingly, the\nGuessWhat?! game rules naturally leads to a rich variety of visually grounded questions. As opposed\nto the VQAv1 dataset, the dataset contains very few commonsense questions that can be answered\nwithout the image.\nIn this paper, we focus on the Oracle task, which is a form of visual question answering in which\nthe answers are limited to yes, no and not applicable. Speci\ufb01cally, the oracle may take as an input\nthe incoming question q, the image I and the target object o\u2217. This object can be described with its\ncategory c, its spatial location and the object crop.\nWe outline here the neural network architecture that was reported in the original GuessWhat?!\npaper [6]. First, we crop the initial image by using the target object bounding box object and rescale\nit to a 224 by 224 square. We then extract the activation of the last convolutional layer after the ReLU\n(stage4) of a pre-trained ResNet-50. We also embed the spatial information of the crop within the\nimage by extracting an 8-dimensional vector of the location of the bounding box\n\n[xmin, ymin, xmax, ymax, xcenter, ycenter, wbox, hbox],\n\n(10)\n\n6\n\n\fTable 3: GuessWhat?! test errors for the Oracle model with different embeddings. Lower is better.\n\nft stage4\n27.48%\n22.68%\n\nFt BN\n27.94%\n22.42%\n\n21.5%\n\nCrop\nCrop + Spatial + Category\nSpatial + Category\n\nRaw features\n\n29.92%\n22.55%\n\nCBN\n25.06%\n19.52%\n\nwhere wbox and hbox denote the width and height of the bounding box, respectively. We convert the\nobject category c into a dense category embedding using a learned look-up table. Finally, we use an\nLSTM to encode the current question q. We then concatenate all embeddings into a single vector\nand feed it as input to a single hidden layer MLP that outputs the \ufb01nal answer distribution using a\nsoftmax layer.\n\n4.3 Baselines\n\nFor both datasets we empirically investigate several modi\ufb01cations to the described architectures. We\nrefer to MODERN when we apply conditional batch normalization to all layers of ResNet-50, as\ndescribed in section 3. To verify that the gains from MODERN are not coming from increased model\ncapacity, we include two baselines with more capacity. The \ufb01rst model \ufb01netunes the layers of stage 4\nof ResNet-50 of our baseline model. This is common practice when we transfer a pretrained network\nto a new task, and we refer it to as Ft Stage 4. We also introduce a novel baseline Ft BN, which\nconsist of \ufb01netuning all \u03b2 and \u03b3 parameters of ResNet-50, while freezing all its weights.\nFor VQA, we report the results of two state-of-the-art architectures, namely, Multimodal Compact\nBilinear pooling network (MCB) [9] (Winner of the VQA challenge 2016) and MUTAN [2]. Both\napproaches employ an (approximate) bilinear pooling mechanism to fuse the language and vision\nembedding by respectively using a random projection and a tensor decomposition. In addition, we\nre-implement and run the MRN model described in Section 4.1. When benchmarking state-of-the-art\nmodels, we train on the training set, proceed early stopping on the validation set and report accuracy\non the test set (test-dev in the case of VQA.)\n\n4.4 Results\n\nVQA We report the best validation accuracy of the outlined methods on the VQA task in Table1.\nNote that we use input images of size 224x224 when we compare MODERN against the baselines (as\nwell as for the ablation study presented in Table 2a. Our initial baseline achieves 58.05% accuracy,\nand we \ufb01nd that \ufb01netuning the last layers (Ft Stage 4) does not improve this performance (56.91%).\nInterestingly, just \ufb01netuning the batch norm parameters (Ft BN) signi\ufb01cantly improves the accuracy\nto 58.98%. We see another signi\ufb01cant performance jump when we condition the batch normalization\non the question input (MODERN), which improves our baseline with almost 2 accuracy points to\n60.82%.\nBecause state-of-the-art models use images of size 448x448, we also include the results of the baseline\narchitecture on these larger images. As seen in Table1, this nearly matches the state of the art results\nwith a 62.15%. As MODERN does not rely on a speci\ufb01c attention mechanism, we then combine our\nproposed method with MRN [13, 14] architecture, and observe that outperforms the state-of-the-art\nMCB model [9] by half a point. Please note that we select MRN [13, 14] over MCB [9] as the latter\nrequires fewer weight parameters and is more stable to train.\nNote that the presented results use a ResNet-50 while other models rely on extracted image em-\nbedding from a ResNet-152. For sake of comparison, we run the baseline models with extracted\nimage embedding from a ResNet-50. Also for the more advanced MRN architecture, we observe\nperformance gains of approximately 2 accuracy points.\n\nGuessWhat?! We report the best test errors for the outlined method on the Oracle task of Guess-\nWhat?! in Table 3. We \ufb01rst compare the results when we only feed the crop of the selected object to\nthe model. We observe the same trend as in VQA. With an error of 25.06%, CBN performs better\nthan than either \ufb01ne-tuning the \ufb01nal block (27.48% error) or the batch-norm parameters (27.94%\n\n7\n\n\f(a) Feature map projection from raw ResNet\n\n(b) Feature map projection from MODERN\n\nFigure 4: t-SNE projection of feature maps (before attention mechanism) of ResNet and MODERN.\nPoints are colored according to the answer type of VQA. Whilst there are no clusters with raw\nfeatures, MODERN successfully modulates the image feature towards speci\ufb01c answer types.\n\nerror), which in turn improve over just using the raw features (29.92% error). Note that the relative\nimprovement (5 error points) for CBN is much bigger for GuessWhat?! than for VQA.\nWe therefore also investigate the performance of the methods when we include the spatial and\ncategory information. We observe that \ufb01netuning the last layers or BN parameters does not improve\nthe performance, while MODERN improves the best reported test error with 2 points to 19.52% error.\n\n4.5 Discussion\n\nBy analyzing the results from both VQA and GuessWhat?! experiments, it is possible to have a better\ninsight regarding MODERN capabilities.\n\nMODERN vs Fine tuning In both experiments, MODERN outperforms Ft BN. Both methods\nupdate the same ResNet parameters so this demonstrates that it is important to condition on the\nlanguage representation. MODERN also outperforms Ft Stage 4 on both tasks which shows that the\nperformance gain of MODERN is not due to the increased model capacity.\n\nConditional embedding In the provided baselines of the Oracle task of GuessWhat?! [6], the\nauthors observed that the best test error (21.5%) is obtained by only providing the object category and\nits spatial location. For this model, including the raw features of the object crop actually deteriorates\nthe performance to 22.55% error. This means that this baseline fails to extract relevant information\nfrom the images which is not in the handcrafted features. Therefore the Oracle can not answer\ncorrectly questions which requires more than the use of spatial information and object category. In\nthe baseline model, the embedding of the crop from a generic ResNet does not help even when we\n\ufb01netune stage 4 or BN. In contrast, applying MODERN helps to better answer questions as the test\nerror drops by 2 points.\n\nAblation study We investigate the impact of only modulating the top layers of a ResNet. We report\nthese results in Table 2. Interestingly, we observe that the performance slowly decreases when we\napply CBN exclusively to later stages. We stress that for best performance it\u2019s important to modulate\nall stages, but if computational resources are limited we recommend to apply it to the two last stages.\n\nVisualizing the representations\nIn order to gain more insight into our proposed fusion mechanism,\nwe compare visualizations of the visual embeddings created by our baseline model and MODERN.\nWe \ufb01rst randomly picked 1000 unique image/question pairs from the validation set of VQA. For\nthe trained MODERN model, we extract image features just before the attention mechanism of\nMODERN, which we will compare with extracted raw ResNet-50 features and \ufb01netune ResNet-50\n(Block4 and batchnorm parameters). We \ufb01rst decrease the dimensionality by average pooling over the\nspatial dimensions of the feature map, and subsequently apply t-SNE [25] to these set of embeddings.\nWe color the points according to the answer type provided by the VQA dataset, and show these\nvisualizations for both models in Fig 4 and Fig 7 in the Appendix B. Interestingly, we observe\nthat all answer types are spread out for raw image features and \ufb01netuned features. In contrast, the\nrepresentations of MODERN are cleanly grouped into three answer types. This demonstrates that\nMODERN successfully disentangles the images representations by answer type which is likely to\n\n8\n\n\fease the later fusion process. While \ufb01netuning models does cluster features, there is no direct link\nbetween those clusters and the answer type. These results indicate that MODERN successfully\nlearns representation that differs from classic \ufb01netuning strategies. In Appendix B, we visualize the\nfeature disentangling process stage by stage. It is possible to spot some sub-clusters in the t-SNE\nrepresentation, as in fact they correspond to image and question pairs which are similar but not\nexplicitly tagged in the VQA dataset. For example, in appendix B the Fig. 6 we highlight pairs where\nthe answer is a color.\n\n5 Related work\n\nMODERN is related to a lot of recent work in VQA[1]. The majority of proposed methods use a\nsimilar computational pipeline introduced by [17, 20]. First, extract high-level image features from a\nImageNet pretrained convnet, while independently processing the question using RNN. Some work\nhas focused on the top level fusing mechanism of the language and visual vectors. For instance, it was\nshown that we can improve upon classic concatenation by an element-wise product [1, 13, 14], Tucker\ndecomposition [2], bilinear pooling [9] or more exotic approaches [18]. Another line of research has\ninvestigated the role of attention mechanisms in VQA [26, 11, 28]. The authors of [11] propose a\nco-attention model over visual and language embeddings, while [28] proposes to stack several spatial\nattention mechanisms. Although an attention mechanism can be thought of as modulating the visual\nfeatures by a language, we stress that such mechanism act on the high-level features. In contrast, our\nwork modulates the visual processing from the very start.\nMODERN is inspired by conditional instance normalization (CIN) [7] that was successfully applied\nto image style transfer. While previous methods transfered one image style per network, [7] showed\nthat up to 32 styles could be compressed into a single network by sharing the convolutional \ufb01lters\nand learning style-speci\ufb01c normalization parameters. There are notable differences with our work.\nFirst, [7] uses a non-differentiable table lookup for the normalization parameters while we propose\na differentiable mapping from the question embedding. Second, we predict a change on the nor-\nmalization parameters of a pretrained convolutional network while keeping the convolutional \ufb01lters\n\ufb01xed. In CIN, all parameters, including the transposed convolutional \ufb01lters, are trained. To the best\nof our knowledge, this is the \ufb01rst paper to conditionally modulate the vision processing using the\nnormalization parameters.\n\n6 Conclusion\n\nIn this paper, we introduce Conditional Batch Normalization (CBN) as a novel fusion mechanism to\nmodulate all layers of a visual processing network. Speci\ufb01cally, we applied CBN to a pre-trained\nResNet, leading to the proposed MODERN architecture. Our approach is motivated by recent\nevidence from neuroscience suggesting that language in\ufb02uences the early stages of visual processing.\nOne of the strengths of MODERN is that it can be incorporated into existing architectures, and our\nexperiments demonstrate that this signi\ufb01cantly improves the baseline models. We also found that it is\nimportant to modulate the entire visual signal to obtain maximum performance gains.\nWhile this paper focuses on text and images, MODERN can be extended to neural architecture\ndealing with other modalities such as sound or video. More broadly, CBN can could also be applied\nto modulate the internal representation of any deep network with respect to any embedding regardless\nof the underlying task. For instance, signal modulation through batch norm parameters may also be\nbene\ufb01cial for reinforcement learning, natural language processing or adversarial training tasks.\n\nAcknowledgements\n\nThe authors would like to acknowledge the stimulating research environment of the SequeL lab.\nWe thank Vincent Dumoulin for helpful discussions about conditional batch normalization. We\nacknowledge the following agencies for research funding and computing support: CHISTERA IGLU\nand CPER Nord-Pas de Calais/FEDER DATA Advanced data science and technologies 2015-2020,\nNSERC, Calcul Qu\u00e9bec, Compute Canada, the Canada Research Chairs and CIFAR. We thank\nNVIDIA for providing access to a DGX-1 machine used in this work.\n\n9\n\n\fReferences\n[1] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, Z. Lawrence, and D. Parikh. Vqa: Visual\n\nquestion answering. In Proc. of ICCV, 2015.\n\n[2] H. Ben-Younes, R. Cad\u00e8ne, N. Thome, and M. Cord. MUTAN: Multimodal Tucker Fusion for\n\nVisual Question Answering. arXiv preprint arXiv:1705.06676, 2017.\n\n[3] B. Boutonnet and G. Lupyan. Words jump-start vision: A label advantage in object recognition.\n\nJournal of Neuroscience, 35(25):9329\u20139335, 2015.\n\n[4] K. Cho, B. Van Merri\u00ebnboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and\nY. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine\ntranslation. In Proc. of EMNLP, 2014.\n\n[5] A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. Moura, D. Parikh, and D. Batra. Visual\n\nDialog. In Proc. of CVPR, 2017.\n\n[6] H. de Vries, F. Strub, S. Chandar, O. Pietquin, H. Larochelle, and A. Courville. GuessWhat?!\n\nVisual object discovery through multi-modal dialogue. In Proc. of CVPR, 2017.\n\n[7] V. Dumoulin, J. Shlens, and M. Kudlur. A Learned Representation For Artistic Style. In Proc.\n\nof ICLR, 2017.\n\n[8] F. Ferreira and M. Tanenhaus. Introduction to the special issue on language\u2013vision interactions.\n\nJournal of Memory and Language, 57(4):455\u2013459, 2007.\n\n[9] A. Fukui, D. Huk Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach. Multimodal\nCompact Bilinear Pooling for Visual Question Answering and Visual Grounding. In Proc. of\nEMNLP, 2016.\n\n[10] S. Hochreiter and J. Schmidhuber. Long short-term memory. In Neural computation, volume 9,\n\npages 1735\u20131780. MIT Press, 1997.\n\n[11] J. Jiasen, J. Yang, D. Batra, and D. Parikh. Hierarchical question-image co-attention for visual\n\nquestion answering. In Proc. of NIPS, 2016.\n\n[12] K. Kaiming, Z. Xiangyu, S. Ren, and J. Sun. Deep residual learning for image recognition. In\n\nProc. of CVPR, 2016.\n\n[13] J. Kim, S. Lee, D. Kwak, M. Heo, J. Kim, J. Ha, and B. Zhang. Multimodal residual learning\n\nfor visual qa. In Proc. of NIPS, 2016.\n\n[14] J. Kim, K. On, J. Kim, J. Ha, and B. Zhang. Hadamard product for low-rank bilinear pooling.\n\nIn Proc. of ICLR, 2017.\n\n[15] P. Kok, M. Failing, and F. de Lange. Prior expectations evoke stimulus templates in the primary\n\nvisual cortex. Journal of Cognitive Neuroscience, 26(7):1546\u20131554, 2014.\n\n[16] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll\u00e1r, and L. Zitnick.\n\nMicrosoft coco: Common objects in context. In Proc of ECCV, 2014.\n\n[17] M. Malinowski, M. Rohrbach, and M. Fritz. Ask your neurons: A neural-based approach to\n\nanswering questions about images. In Proc. of ICCV, 2015.\n\n[18] M. Malinowski, M. Rohrbach, and M. Fritz. Ask your neurons: A deep learning approach to\n\nvisual question answering. arXiv preprint arXiv:1605.02697, 2016.\n\n[19] J. Pennington, R. Socher, and C. Manning. Glove: Global Vectors for Word Representation. In\n\nProc. of EMNLP, 2014.\n\n[20] M. Ren, R. Kiros, and R. Zemel. Exploring models and data for image question answering. In\n\nProc. of NIPS, 2015.\n\n[21] I. Sergey and S. Christian. Batch Normalization: Accelerating Deep Network Training by\n\nReducing Internal Covariate Shift. In Proc. of ICML, 2015.\n\n10\n\n\f[22] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image\n\nrecognition. 2015.\n\n[23] G. Yashand K. Tejas, S. Douglas, Dhruv B, and P. Devi. Making the V in VQA matter: Elevating\n\nthe role of image understanding in Visual Question Answering. In Proc. of CVPR, 2017.\n\n[24] G. Thierry, P. Athanasopoulos, A. Wiggett, B. Dering, and JR. Kuipers. Unconscious effects\nof language-speci\ufb01c terminology on preattentive color perception. PNAS, 106(11):4567\u20134570,\n2009.\n\n[25] L. Maaten van G. der and Hinton. Visualizing data using t-sne. JMLR, 9(Nov):2579\u20132605,\n\n2008.\n\n[26] H. Xu and K. Saenko. Ask, attend and answer: Exploring question-guided spatial attention for\n\nvisual question answering. In Proc. of ECCV, 2015.\n\n[27] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio. Show,\nattend and tell: Neural image caption generation with visual attention. In Proc. of ICML, 2015.\n\n[28] Z. Yang, X. He, J. Gao, and L. Deng A. Smola. Stacked attention networks for image question\n\nanswering. In Proc. of CVPR, 2016.\n\n11\n\n\f", "award": [], "sourceid": 3302, "authors": [{"given_name": "Harm", "family_name": "de Vries", "institution": "Universit\u00e9 de Montr\u00e9al"}, {"given_name": "Florian", "family_name": "Strub", "institution": "University of Lille"}, {"given_name": "Jeremie", "family_name": "Mary", "institution": "INRIA / Univ. Lille"}, {"given_name": "Hugo", "family_name": "Larochelle", "institution": "Google Brain"}, {"given_name": "Olivier", "family_name": "Pietquin", "institution": "DeepMind"}, {"given_name": "Aaron", "family_name": "Courville", "institution": "U. Montreal"}]}