Gone Fishing: Neural Active Learning with Fisher Embeddings
  There is an increasing need for effective active learning algorithms that are
compatible with deep neural networks. This paper motivates and revisits a
classic, Fisher-based active selection objective, and proposes BAIT, a
practical, tractable, and high-performing algorithm that makes it viable for
use with neural models. BAIT draws inspiration from the theoretical analysis of
maximum likelihood estimators (MLE) for parametric models. It selects batches
of samples by optimizing a bound on the MLE error in terms of the Fisher
information, which we show can be implemented efficiently at scale by
exploiting linear-algebraic structure especially amenable to execution on
modern hardware. Our experiments demonstrate that BAIT outperforms the previous
state of the art on both classification and regression problems, and is
flexible enough to be used with a variety of model architectures.
