Visio-Linguistic Brain Encoding
  Enabling effective brain-computer interfaces requires understanding how the
human brain encodes stimuli across modalities such as visual, language (or
text), etc. Brain encoding aims at constructing fMRI brain activity given a
stimulus. There exists a plethora of neural encoding models which study brain
encoding for single mode stimuli: visual (pretrained CNNs) or text (pretrained
language models). Few recent papers have also obtained separate visual and text
representation models and performed late-fusion using simple heuristics.
However, previous work has failed to explore: (a) the effectiveness of image
Transformer models for encoding visual stimuli, and (b) co-attentive
multi-modal modeling for visual and text reasoning. In this paper, we
systematically explore the efficacy of image Transformers (ViT, DEiT, and BEiT)
and multi-modal Transformers (VisualBERT, LXMERT, and CLIP) for brain encoding.
Extensive experiments on two popular datasets, BOLD5000 and Pereira, provide
the following insights. (1) To the best of our knowledge, we are the first to
investigate the effectiveness of image and multi-modal Transformers for brain
encoding. (2) We find that VisualBERT, a multi-modal Transformer, significantly
outperforms previously proposed single-mode CNNs, image Transformers as well as
other previously proposed multi-modal models, thereby establishing new
state-of-the-art. The supremacy of visio-linguistic models raises the question
of whether the responses elicited in the visual regions are affected implicitly
by linguistic processing even when passively viewing images. Future fMRI tasks
can verify this computational insight in an appropriate experimental setting.
