How does unlabeled data improve generalization in self-training? A
  one-hidden-layer theoretical analysis
  Self-training, a semi-supervised learning algorithm, leverages a large amount
of unlabeled data to improve learning when the labeled data are limited.
Despite empirical successes, its theoretical characterization remains elusive.
To the best of our knowledge, this work establishes the first theoretical
analysis for the known iterative self-training paradigm and proves the benefits
of unlabeled data in both training convergence and generalization ability. To
make our theoretical analysis feasible, we focus on the case of
one-hidden-layer neural networks. However, theoretical understanding of
iterative self-training is non-trivial even for a shallow neural network. One
of the key challenges is that existing neural network landscape analysis built
upon supervised learning no longer holds in the (semi-supervised) self-training
paradigm. We address this challenge and prove that iterative self-training
converges linearly with both convergence rate and generalization accuracy
improved in the order of $1/\sqrt{M}$, where $M$ is the number of unlabeled
samples. Experiments from shallow neural networks to deep neural networks are
also provided to justify the correctness of our established theoretical
insights on self-training.
