How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the
  Winograd Schema Challenge and SWAG
  Recent studies have significantly improved the state-of-the-art on
common-sense reasoning (CSR) benchmarks like the Winograd Schema Challenge
(WSC) and SWAG. The question we ask in this paper is whether improved
performance on these benchmarks represents genuine progress towards
common-sense-enabled systems. We make case studies of both benchmarks and
design protocols that clarify and qualify the results of previous work by
analyzing threats to the validity of previous experimental designs. Our
protocols account for several properties prevalent in common-sense benchmarks
including size limitations, structural regularities, and variable instance
difficulty.
