Towards Understanding the Behaviors of Optimal Deep Active Learning
  Algorithms
  Active learning (AL) algorithms may achieve better performance with fewer
data because the model guides the data selection process. While many algorithms
have been proposed, there is little study on what the optimal AL algorithm
looks like, which would help researchers understand where their models fall
short and iterate on the design. In this paper, we present a simulated
annealing algorithm to search for this optimal oracle and analyze it for
several tasks. We present qualitative and quantitative insights into the
behaviors of this oracle, comparing and contrasting them with those of various
heuristics. Moreover, we are able to consistently improve the heuristics using
one particular insight. We hope that our findings can better inform future
active learning research. The code is available at
https://github.com/YilunZhou/optimal-active-learning.
