Concept-Centric Visual Turing Tests for Method Validation
  Recent advances in machine learning for medical imaging have led to
impressive increases in model complexity and overall capabilities. However, the
ability to discern the precise information a machine learning method is using
to make decisions has lagged behind and it is often unclear how these
performances are in fact achieved. Conventional evaluation metrics that reduce
method performance to a single number or a curve only provide limited insights.
Yet, systems used in clinical practice demand thorough validation that such
crude characterizations miss. To this end, we present a framework to evaluate
classification methods based on a number of interpretable concepts that are
crucial for a clinical task. Our approach is inspired by the Turing Test
concept and how to devise a test that adaptively questions a method for its
ability to interpret medical images. To do this, we make use of a Twenty
Questions paradigm whereby we use a probabilistic model to characterize the
method's capacity to grasp task-specific concepts, and we introduce a strategy
to sequentially query the method according to its previous answers. The results
show that the probabilistic model is able to expose both the dataset's and the
method's biases, and can be used to reduced the number of queries needed for
confident performance evaluation.
