A logical-based corpus for cross-lingual evaluation
  At present, different deep learning models are presenting high accuracy on
popular inference datasets such as SNLI, MNLI, and SciTail. However, there are
different indicators that those datasets can be exploited by using some simple
linguistic patterns. This fact poses difficulties to our understanding of the
actual capacity of machine learning models to solve the complex task of textual
inference. We propose a new set of syntactic tasks focused on contradiction
detection that require specific capacities over linguistic logical forms such
as: Boolean coordination, quantifiers, definite description, and counting
operators. We evaluate two kinds of deep learning models that implicitly
exploit language structure: recurrent models and the Transformer network BERT.
We show that although BERT is clearly more efficient to generalize over most
logical forms, there is space for improvement when dealing with counting
operators. Since the syntactic tasks can be implemented in different languages,
we show a successful case of cross-lingual transfer learning between English
and Portuguese.
