BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models
  Identify Analogies?
  Analogies play a central role in human commonsense reasoning. The ability to
recognize analogies such as "eye is to seeing what ear is to hearing",
sometimes referred to as analogical proportions, shape how we structure
knowledge and understand language. Surprisingly, however, the task of
identifying such analogies has not yet received much attention in the language
model era. In this paper, we analyze the capabilities of transformer-based
language models on this unsupervised task, using benchmarks obtained from
educational settings, as well as more commonly used datasets. We find that
off-the-shelf language models can identify analogies to a certain extent, but
struggle with abstract and complex relations, and results are highly sensitive
to model architecture and hyperparameters. Overall the best results were
obtained with GPT-2 and RoBERTa, while configurations using BERT were not able
to outperform word embedding models. Our results raise important questions for
future work about how, and to what extent, pre-trained language models capture
knowledge about abstract semantic relations.
