An empirical study of domain-agnostic semi-supervised learning via
  energy-based models: joint-training and pre-training
  A class of recent semi-supervised learning (SSL) methods heavily rely on
domain-specific data augmentations. In contrast, generative SSL methods involve
unsupervised learning based on generative models by either joint-training or
pre-training, and are more appealing from the perspective of being
domain-agnostic, since they do not inherently require data augmentations.
Joint-training estimates the joint distribution of observations and labels,
while pre-training is taken over observations only. Recently, energy-based
models (EBMs) have achieved promising results for generative modeling.
Joint-training via EBMs for SSL has been explored with encouraging results
across different data modalities. In this paper, we make two contributions.
First, we explore pre-training via EBMs for SSL and compare it to
joint-training. Second, a suite of experiments are conducted over domains of
image classification and natural language labeling to give a realistic whole
picture of the performances of EBM based SSL methods. It is found that
joint-training EBMs outperform pre-training EBMs marginally but nearly
consistently.
