ORCHARD: A Benchmark For Measuring Systematic Generalization of
  Multi-Hierarchical Reasoning
  The ability to reason with multiple hierarchical structures is an attractive
and desirable property of sequential inductive biases for natural language
processing. Do the state-of-the-art Transformers and LSTM architectures
implicitly encode for these biases? To answer this, we propose ORCHARD, a
diagnostic dataset for systematically evaluating hierarchical reasoning in
state-of-the-art neural sequence models. While there have been prior evaluation
frameworks such as ListOps or Logical Inference, our work presents a novel and
more natural setting where our models learn to reason with multiple explicit
hierarchical structures instead of only one, i.e., requiring the ability to do
both long-term sequence memorizing, relational reasoning while reasoning with
hierarchical structure. Consequently, backed by a set of rigorous experiments,
we show that (1) Transformer and LSTM models surprisingly fail in systematic
generalization, and (2) with increased references between hierarchies,
Transformer performs no better than random.
