Semi-Supervised Empirical Risk Minimization: Using unlabeled data to
  improve prediction
  We present a general methodology for using unlabeled data to design semi
supervised learning (SSL) variants of the Empirical Risk Minimization (ERM)
learning process. Focusing on generalized linear regression, we analyze of the
effectiveness of our SSL approach in improving prediction performance. The key
ideas are carefully considering the null model as a competitor, and utilizing
the unlabeled data to determine signal-noise combinations where SSL outperforms
both supervised learning and the null model. We then use SSL in an adaptive
manner based on estimation of the signal and noise. In the special case of
linear regression with Gaussian covariates, we prove that the non-adaptive SSL
version is in fact not capable of improving on both the supervised estimator
and the null model simultaneously, beyond a negligible O(1/n) term. On the
other hand, the adaptive model presented in this work, can achieve a
substantial improvement over both competitors simultaneously, under a variety
of settings. This is shown empirically through extensive simulations, and
extended to other scenarios, such as non-Gaussian covariates, misspecified
linear regression, or generalized linear regression with non-linear link
functions.
