Active Surrogate Estimators: An Active Learning Approach to
  Label-Efficient Model Evaluation
  We propose Active Surrogate Estimators (ASEs), a new method for
label-efficient model evaluation. Evaluating model performance is a challenging
and important problem when labels are expensive. ASEs address this active
testing problem using a surrogate-based estimation approach, whereas previous
methods have focused on Monte Carlo estimates. ASEs actively learn the
underlying surrogate, and we propose a novel acquisition strategy, XWING, that
tailors this learning to the final estimation task. We find that ASEs offer
greater label-efficiency than the current state-of-the-art when applied to
challenging model evaluation problems for deep neural networks. We further
theoretically analyze ASEs' errors.
