Measuring Discrimination to Boost Comparative Testing for Multiple Deep
  Learning Models
  The boom of DL technology leads to massive DL models built and shared, which
facilitates the acquisition and reuse of DL models. For a given task, we
encounter multiple DL models available with the same functionality, which are
considered as candidates to achieve this task. Testers are expected to compare
multiple DL models and select the more suitable ones w.r.t. the whole testing
context. Due to the limitation of labeling effort, testers aim to select an
efficient subset of samples to make an as precise rank estimation as possible
for these models. To tackle this problem, we propose Sample Discrimination
based Selection (SDS) to select efficient samples that could discriminate
multiple models, i.e., the prediction behaviors (right/wrong) of these samples
would be helpful to indicate the trend of model performance. To evaluate SDS,
we conduct an extensive empirical study with three widely-used image datasets
and 80 real world DL models. The experimental results show that, compared with
state-of-the-art baseline methods, SDS is an effective and efficient sample
selection method to rank multiple DL models.
