A Methodology for Creating Question Answering Corpora Using Inverse Data
  Annotation
  In this paper, we introduce a novel methodology to efficiently construct a
corpus for question answering over structured data. For this, we introduce an
intermediate representation that is based on the logical query plan in a
database called Operation Trees (OT). This representation allows us to invert
the annotation process without losing flexibility in the types of queries that
we generate. Furthermore, it allows for fine-grained alignment of query tokens
to OT operations. In our method, we randomly generate OTs from a context-free
grammar. Afterwards, annotators have to write the appropriate natural language
question that is represented by the OT. Finally, the annotators assign the
tokens to the OT operations. We apply the method to create a new corpus OTTA
(Operation Trees and Token Assignment), a large semantic parsing corpus for
evaluating natural language interfaces to databases. We compare OTTA to Spider
and LC-QuaD 2.0 and show that our methodology more than triples the annotation
speed while maintaining the complexity of the queries. Finally, we train a
state-of-the-art semantic parsing model on our data and show that our corpus is
a challenging dataset and that the token alignment can be leveraged to increase
the performance significantly.
