Exploring Self-Supervised Regularization for Supervised and
  Semi-Supervised Learning
  Recent advances in semi-supervised learning have shown tremendous potential
in overcoming a major barrier to the success of modern machine learning
algorithms: access to vast amounts of human-labeled training data. Previous
algorithms based on consistency regularization can harness the abundance of
unlabeled data to produce impressive results on a number of semi-supervised
benchmarks, approaching the performance of strong supervised baselines using
only a fraction of the available labeled data. In this work, we challenge the
long-standing success of consistency regularization by introducing
self-supervised regularization as the basis for combining semantic feature
representations from unlabeled data. We perform extensive comparative
experiments to demonstrate the effectiveness of self-supervised regularization
for supervised and semi-supervised image classification on SVHN, CIFAR-10, and
CIFAR-100 benchmark datasets. We present two main results: (1) models augmented
with self-supervised regularization significantly improve upon traditional
supervised classifiers without the need for unlabeled data; (2) together with
unlabeled data, our models yield semi-supervised performance competitive with,
and in many cases exceeding, prior state-of-the-art consistency baselines.
Lastly, our models have the practical utility of being efficiently trained
end-to-end and require no additional hyper-parameters to tune for optimal
performance beyond the standard set for training neural networks. Reference
code and data are available at https://github.com/vuptran/sesemi
