Practical Assessment of Generalization Performance Robustness for Deep
  Networks via Contrastive Examples
  Training images with data transformations have been suggested as contrastive
examples to complement the testing set for generalization performance
evaluation of deep neural networks (DNNs). In this work, we propose a practical
framework ContRE (The word "contre" means "against" or "versus" in French.)
that uses Contrastive examples for DNN geneRalization performance Estimation.
Specifically, ContRE follows the assumption in contrastive learning that robust
DNN models with good generalization performance are capable of extracting a
consistent set of features and making consistent predictions from the same
image under varying data transformations. Incorporating with a set of
randomized strategies for well-designed data transformations over the training
set, ContRE adopts classification errors and Fisher ratios on the generated
contrastive examples to assess and analyze the generalization performance of
deep models in complement with a testing set. To show the effectiveness and the
efficiency of ContRE, extensive experiments have been done using various DNN
models on three open source benchmark datasets with thorough ablation studies
and applicability analyses. Our experiment results confirm that (1) behaviors
of deep models on contrastive examples are strongly correlated to what on the
testing set, and (2) ContRE is a robust measure of generalization performance
complementing to the testing set in various settings.
