Relieving the Plateau: Active Semi-Supervised Learning for a Better
  Landscape
  Deep learning (DL) relies on massive amounts of labeled data, and improving
its labeled sample-efficiency remains one of the most important problems since
its advent. Semi-supervised learning (SSL) leverages unlabeled data that are
more accessible than their labeled counterparts. Active learning (AL) selects
unlabeled instances to be annotated by a human-in-the-loop in hopes of better
performance with less labeled data. Given the accessible pool of unlabeled data
in pool-based AL, it seems natural to use SSL when training and AL to update
the labeled set; however, algorithms designed for their combination remain
limited. In this work, we first prove that convergence of gradient descent on
sufficiently wide ReLU networks can be expressed in terms of their Gram matrix'
eigen-spectrum. Equipped with a few theoretical insights, we propose
convergence rate control (CRC), an AL algorithm that selects unlabeled data to
improve the problem conditioning upon inclusion to the labeled set, by
formulating an acquisition step in terms of improving training dynamics.
Extensive experiments show that SSL algorithms coupled with CRC can achieve
high performance using very few labeled data.
