DQI: A Guide to Benchmark Evaluation
  A `state of the art' model A surpasses humans in a benchmark B, but fails on
similar benchmarks C, D, and E. What does B have that the other benchmarks do
not? Recent research provides the answer: spurious bias. However, developing A
to solve benchmarks B through E does not guarantee that it will solve future
benchmarks. To progress towards a model that `truly learns' an underlying task,
we need to quantify the differences between successive benchmarks, as opposed
to existing binary and black-box approaches. We propose a novel approach to
solve this underexplored task of quantifying benchmark quality by debuting a
data quality metric: DQI.
