FAME: Feature-Based Adversarial Meta-Embeddings for Robust Input
  Representations
  Combining several embeddings typically improves performance in downstream
tasks as different embeddings encode different information. It has been shown
that even models using embeddings from transformers still benefit from the
inclusion of standard word embeddings. However, the combination of embeddings
of different types and dimensions is challenging. As an alternative to
attention-based meta-embeddings, we propose feature-based adversarial
meta-embeddings (FAME) with an attention function that is guided by features
reflecting word-specific properties, such as shape and frequency, and show that
this is beneficial to handle subword-based embeddings. In addition, FAME uses
adversarial training to optimize the mappings of differently-sized embeddings
to the same space. We demonstrate that FAME works effectively across languages
and domains for sequence labeling and sentence classification, in particular in
low-resource settings. FAME sets the new state of the art for POS tagging in 27
languages, various NER settings and question classification in different
domains.
