Low-Dimensional Structure in the Space of Language Representations is
  Reflected in Brain Responses
  How related are the representations learned by neural language models,
translation models, and language tagging tasks? We answer this question by
adapting an encoder-decoder transfer learning method from computer vision to
investigate the structure among 100 different feature spaces extracted from
hidden representations of various networks trained on language tasks. This
method reveals a low-dimensional structure where language models and
translation models smoothly interpolate between word embeddings, syntactic and
semantic tasks, and future word embeddings. We call this low-dimensional
structure a language representation embedding because it encodes the
relationships between representations needed to process language for a variety
of NLP tasks. We find that this representation embedding can predict how well
each individual feature space maps to human brain responses to natural language
stimuli recorded using fMRI. Additionally, we find that the principal dimension
of this structure can be used to create a metric which highlights the brain's
natural language processing hierarchy. This suggests that the embedding
captures some part of the brain's natural language representation structure.
