Taming Overconfident Prediction on Unlabeled Data from Hindsight
  Minimizing prediction uncertainty on unlabeled data is a key factor to
achieve good performance in semi-supervised learning (SSL). The prediction
uncertainty is typically expressed as the \emph{entropy} computed by the
transformed probabilities in output space. Most existing works distill
low-entropy prediction by either accepting the determining class (with the
largest probability) as the true label or suppressing subtle predictions (with
the smaller probabilities). Unarguably, these distillation strategies are
usually heuristic and less informative for model training. From this
discernment, this paper proposes a dual mechanism, named ADaptive Sharpening
(\ADS), which first applies a soft-threshold to adaptively mask out determinate
and negligible predictions, and then seamlessly sharpens the informed
predictions, distilling certain predictions with the informed ones only. More
importantly, we theoretically analyze the traits of \ADS by comparing with
various distillation strategies. Numerous experiments verify that \ADS
significantly improves the state-of-the-art SSL methods by making it a plug-in.
Our proposed \ADS forges a cornerstone for future distillation-based SSL
research.
