Probing Contextualized Sentence Representations with Visual Awareness
  We present a universal framework to model contextualized sentence
representations with visual awareness that is motivated to overcome the
shortcomings of the multimodal parallel data with manual annotations. For each
sentence, we first retrieve a diversity of images from a shared cross-modal
embedding space, which is pre-trained on a large-scale of text-image pairs.
Then, the texts and images are respectively encoded by transformer encoder and
convolutional neural network. The two sequences of representations are further
fused by a simple and effective attention layer. The architecture can be easily
applied to text-only natural language processing tasks without manually
annotating multimodal parallel corpora. We apply the proposed method on three
tasks, including neural machine translation, natural language inference and
sequence labeling and experimental results verify the effectiveness.
