Brain2Word: Decoding Brain Activity for Language Generation
  Brain decoding, understood as the process of mapping brain activities to the
stimuli that generated them, has been an active research area in the last
years. In the case of language stimuli, recent studies have shown that it is
possible to decode fMRI scans into an embedding of the word a subject is
reading. However, such word embeddings are designed for natural language
processing tasks rather than for brain decoding. Therefore, they limit our
ability to recover the precise stimulus. In this work, we propose to directly
classify an fMRI scan, mapping it to the corresponding word within a fixed
vocabulary. Unlike existing work, we evaluate on scans from previously unseen
subjects. We argue that this is a more realistic setup and we present a model
that can decode fMRI data from unseen subjects. Our model achieves 5.22% Top-1
and 13.59% Top-5 accuracy in this challenging task, significantly outperforming
all the considered competitive baselines. Furthermore, we use the decoded words
to guide language generation with the GPT-2 model. This way, we advance the
quest for a system that translates brain activities into coherent text.
