What BERT Sees: Cross-Modal Transfer for Visual Question Generation
  Pre-trained language models have recently contributed to significant advances
in NLP tasks. Recently, multi-modal versions of BERT have been developed, using
heavy pre-training relying on vast corpora of aligned textual and image data,
primarily applied to classification tasks such as VQA. In this paper, we are
interested in evaluating the visual capabilities of BERT out-of-the-box, by
avoiding pre-training made on supplementary data. We choose to study Visual
Question Generation, a task of great interest for grounded dialog, that enables
to study the impact of each modality (as input can be visual and/or textual).
Moreover, the generation aspect of the task requires an adaptation since BERT
is primarily designed as an encoder. We introduce BERT-gen, a BERT-based
architecture for text generation, able to leverage on either mono- or multi-
modal representations. The results reported under different configurations
indicate an innate capacity for BERT-gen to adapt to multi-modal data and text
generation, even with few data available, avoiding expensive pre-training. The
proposed model obtains substantial improvements over the state-of-the-art on
two established VQG datasets.
